Skip to content

Update logfilter.py #1426

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jangjangji wants to merge 1 commit into
base: main
Choose a base branch
from
+5 −1
Open

Update logfilter.py #1426

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion gpMgmt/bin/gppylib/logfilter.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -278,13 +278,17 @@ def __next__(self):
item = next(self.source)
#we need to make a minor format change to the log level field so that
# our single regex will match both.
item[16] = item[16] + ": "
# Check if item has enough columns before accessing index 16
if isinstance(item, list) and len(item) > 16:
item[16] = item[16] + ": "
Comment on lines +282 to +283
Copy link
Contributor

@gfphoenix78 gfphoenix78 Nov 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When is the condition false? Is it correct to ignore the else branch?

Copy link
Author

@jangjangji jangjangji Nov 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

### When is the condition false?

The condition is false when len(item) <= 16, which occurs when the CSV reader incorrectly parses lines containing multi-line strings (e.g., SQL
statements with newlines).

While our CSV files follow the standard 23-column PostgreSQL csvlog format, some log entries contain multi-line quoted strings that cause the CSV
reader to return incomplete rows with fewer than 17 columns.

Is it correct to ignore the else branch?

Yes. The item[16] + ": " transformation is optional formatting to help regex pattern matching - not a required operation.

Without this check, a single malformed line causes IndexError, which terminates the entire iterator silently (treated as StopIteration),
resulting in 0 output despite reading thousands of lines.

By skipping the transformation for malformed lines:

  • Valid lines (23 columns) → processed normally with the transformation
  • Malformed lines (< 17 columns) → passed through unchanged
  • Downstream filters naturally handle/discard non-matching entries

This allows processing to continue instead of failing silently on the first malformed line.

Copy link
Author

@jangjangji jangjangji Mar 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gfphoenix78 Sorry for the late reply. I've just added my response above. Please take a look when you have time.


self.buffer.truncate(0)
self.buffer.seek(0)
self.writer.writerow(item)

return self.buffer.getvalue()


#------------------------------- Spying --------------------------------

class Count(object):
Expand Down
Loading