feat(isthmus): add PostgreSQL TPC-H integration testing

[nielspardon]

This PR adds integration testing for isthmus against a PostgreSQL database.

• it adds TPC-H data generated with scale factor 0.01 aka tiny *.tbl files
• it uses testcontainers to automatically spin up a PostgreSQL container during testing
• populates the PostgreSQL DB by running the DDL statements and loading the data from the *.tbl files
• runs a reference SQL and a SQL generated from the Substrait plan against the PostgreSQL database and checks whether both SQLs produce the same records
• I had to create copies of the DDL statements and the TPC-H queries for PostgreSQL since there are smaller syntax differences vs the copies we currently have in the queries subfolder e.g.
  • PostgreSQL expects interval '1 day' instead of interval '1' day(3).
  • identifiers in Substrait plans are essentially handled as case-sensitive so I had to uppercase and quote identifiers in the reference SQLs and the DDL statements
• the good news all but TPC-H query 21 are producing the same results based on the testing method I'm using. just need to figure out what's going wrong with this query

This is currently a draft since I want to get some early feedback. What I'm still planning to do:

• add javadocs for the new code
• Q: should we always run the integration tests or should we run them in a separate Gradle task?
• I also want to enable this for the TPC-DS in a follow-up PR and then see whether we can extend this more databases supported by testcontainers

[benbellick]

I am working on a fuller response here, but just wanted to see if you were aware of the existence of this related-seeming repository?

[nielspardon]

I am working on a fuller response here, but just wanted to see if you were aware of the existence of this related-seeming repository?

I am aware of the repo. The problem with the consumer-testing repo is that by design it is a centralized approach and hence difficult to maintain. There is currently no active maintenance of it. I would suggest to not do the consumer testing centrally but to provide a Substrait testkit that integrations can test against in a decentralized setup. This brings the testing closer to the code being tested allowing for shorter cycles. This e.g. what a project like dbt does with their many adapters.

@andrew-coleman has been looking into updating the consumer-testing repo to a more recent spec version and it led him down a rabbit hole of many of the currently tested consumers not having updated to the latest spec version. In some cases ending up with seemingly unmaintained consumers like e.g. Apache Ibis.

[benbellick]

Have these tests caught any broken behavior? I am apprehensive to introduce 90,000 lines of code in this PR. Is there some way to make these test files dynamically fetched from their source? E.g. an existing java package which wraps these static files, or a URL from which they can be cloned? If I understand correctly, these files are intended to never be modified.

[nielspardon]

Have these tests caught any broken behavior?

yes, see above:

• the good news all but TPC-H query 21 are producing the same results based on the testing method I'm using. just need to figure out what's going wrong with this query

--

I am apprehensive to introduce 90,000 lines of code in this PR.

most of it is not code but test data

--

Is there some way to make these test files dynamically fetched from their source? E.g. an existing java package which wraps these static files, or a URL from which they can be cloned? If I understand correctly, these files are intended to never be modified.

There is no existing source to fetch them from. There is no official TPC dataset but TPC provides a data generator with which one can generate the test data. What we could do is we can move the test data into the consumer-testing repo and transform the consumer-testing repo into a testkit repo instead of a centralized testing repository which performs the testing.