Elasticsearch search backend implementation

AGENTS.md

@@ -0,0 +1,63 @@
+# Architecture
+
+## Frontend
+
+Use TAB instead of spaces (frontend only).
+
+There are two similar frontends in the Modrinth monorepo, the website (apps/frontend) and the app frontend (apps/app-frontend).
+
+Both use Tailwind v3, and their respective configs can be seen at `tailwind.config.ts` and `tailwind.config.js` respectively.
+
+Both utilize shared and common components from `@modrinth/ui` which can be found at `packages/ui`, and stylings from `@modrinth/assets` which can be found at `packages/assets`.
+
+Both can utilize icons from `@modrinth/assets`, which are automatically generated based on what's available within the `icons` folder of the `packages/assets` directory. You can see the generated icons list in `generated-icons.ts`.
+
+Both have access to our dependency injection framework, examples as seen in `packages/ui/src/providers/`. Ideally any state which is shared between a page and it's subpages should be shared using this dependency injection framework.
+
+### Website (apps/frontend)
+
+Before a pull request can be opened for the website, run `pnpm prepr:frontend:web` from the root folder, otherwise CI will fail.
+
+To run a development version of the frontend, you must first copy over the relevant `.env` template file (prod, staging or local, usually prod) within the `apps/frontend` folder into `apps/frontend/.env`. Then you can run the frontend by running `pnpm web:dev` in the root folder.
+
+### App Frontend (apps/app-frontend)
+
+Before a pull request can be opened for the app frontend, run `pnpm prepr:frontend:app` from the root folder, otherwise CI will fail.
+
+To run a development version of the app frontend, you must first copy over the relevant `.env` template file (prod, staging or local, usually prod) within `packages/app-lib` into `packages/app-lib/.env`. Then you must run the app itself by running `pnpm app:dev` in the root folder.
+
+### Localization
+
+Refer to `.github/instructions/i18n-convert.instructions.md` if the user asks you to perform any i18n conversion work on a component, set of components, pages or sets of pages.
+
+## Labrinth
+
+Labrinth is the backend API service for Modrinth. It can be ran by changing directory into `apps/labrinth` and running `cargo run -p labrinth`. This MUST be run in the `apps/labrinth` directory, as otherwise the `.env` will not be read. If some environment variables are missing, their default values can be found in `.env.local`. The backend runs on `localhost:8000` by default - you can `curl http://localhost:8000` to test requests.
+
+### Testing
+
+Before a pull request can be opened, run `cargo clippy -p labrinth --all-targets` and make sure there are ZERO warnings, otherwise CI will fail.
+
+Use `cargo test -p labrinth --all-targets` to test your changes. All tests must pass, otherwise CI will fail.
+
+To prepare the sqlx cache, cd into `apps/labrinth` and run `cargo sqlx prepare`. Make sure to NEVER run `cargo sqlx prepare --workspace`.
+
+Read the root `docker-compose.yml` to see what running services are available while developing. Use `docker exec` or `podman exec` to access these services.
+
+When the user refers to "performing pre-PR checks", do the following:
+
+- Run clippy as described above
+- DO NOT run tests unless explicitly requested (they take a long time)
+- Prepare the sqlx cache
+
+### Clickhouse
+
+Use `docker exec labrinth-clickhouse clickhouse-client` to access the Clickhouse instance. We use the `staging_ariadne` database to store data in testing.
+
+### Postgres
+
+Use `docker exec labrinth-postgres psql -U labrinth -d labrinth -c "SELECT 1"` to access the PostgreSQL instance, replacing the `SELECT 1` with your query.
+
+# Guidelines
+
+- Do not create new non-source code files (e.g. Bash scripts, SQL scripts) unless explicitly prompted to.

CLAUDE.md

@@ -1,9 +1,9 @@
 # Architecture
 
-Use TAB instead of spaces.
-
 ## Frontend
 
+Use TAB instead of spaces (frontend only).
+
 There are two similar frontends in the Modrinth monorepo, the website (apps/frontend) and the app frontend (apps/app-frontend).
 
 Both use Tailwind v3, and their respective configs can be seen at `tailwind.config.ts` and `tailwind.config.js` respectively.
@@ -32,7 +32,7 @@ Refer to `.github/instructions/i18n-convert.instructions.md` if the user asks yo
 
 ## Labrinth
 
-Labrinth is the backend API service for Modrinth.
+Labrinth is the backend API service for Modrinth. It can be ran by changing directory into `apps/labrinth` and running `cargo run -p labrinth`. This MUST be run in the `apps/labrinth` directory, as otherwise the `.env` will not be read. If some environment variables are missing, their default values can be found in `.env.local`. The backend runs on `localhost:8000` by default - you can `curl http://localhost:8000` to test requests.
 
 ### Testing
 
@@ -42,7 +42,7 @@ Use `cargo test -p labrinth --all-targets` to test your changes. All tests must
 
 To prepare the sqlx cache, cd into `apps/labrinth` and run `cargo sqlx prepare`. Make sure to NEVER run `cargo sqlx prepare --workspace`.
 
-Read the root `docker-compose.yml` to see what running services are available while developing. Use `docker exec` to access these services.
+Read the root `docker-compose.yml` to see what running services are available while developing. Use `docker exec` or `podman exec` to access these services.
 
 When the user refers to "performing pre-PR checks", do the following:
 

Cargo.toml

@@ -69,6 +69,7 @@ dotenv-build = "0.1.1"
 dotenvy = "0.15.7"
 dunce = "1.0.5"
 either = "1.15.0"
+elasticsearch = "9.1.0-alpha.1"
 encoding_rs = "0.8.35"
 enumset = "1.1.10"
 eyre = "0.6.12"

apps/labrinth/.env.docker-compose

@@ -16,9 +16,14 @@ DATABASE_URL=postgresql://labrinth:labrinth@labrinth-postgres/labrinth
 DATABASE_MIN_CONNECTIONS=0
 DATABASE_MAX_CONNECTIONS=16
 
+SEARCH_BACKEND=meilisearch
 MEILISEARCH_READ_ADDR=http://localhost:7700
 MEILISEARCH_WRITE_ADDRS=http://localhost:7700
 MEILISEARCH_KEY=modrinth
+ELASTICSEARCH_URL=http://localhost:9200
+ELASTICSEARCH_INDEX_PREFIX=labrinth
+ELASTICSEARCH_USERNAME=elastic
+ELASTICSEARCH_PASSWORD=elastic
 
 REDIS_URL=redis://labrinth-redis
 REDIS_MIN_CONNECTIONS=0

apps/labrinth/.env.local

@@ -16,16 +16,29 @@ DATABASE_URL=postgresql://labrinth:labrinth@localhost/labrinth
 DATABASE_MIN_CONNECTIONS=0
 DATABASE_MAX_CONNECTIONS=16
 
+SEARCH_BACKEND=meilisearch
+
+# Meilisearch configuration
 MEILISEARCH_READ_ADDR=http://localhost:7700
 MEILISEARCH_WRITE_ADDRS=http://localhost:7700
 # 5 minutes in milliseconds
 SEARCH_OPERATION_TIMEOUT=300000
 
+ELASTICSEARCH_URL=http://localhost:9200
+ELASTICSEARCH_INDEX_PREFIX=labrinth
+
 # # For a sharded Meilisearch setup (sharded-meilisearch docker compose profile)
 # MEILISEARCH_READ_ADDR=http://localhost:7710
 # MEILISEARCH_WRITE_ADDRS=http://localhost:7700,http://localhost:7701
 
 MEILISEARCH_KEY=modrinth
+MEILISEARCH_META_NAMESPACE=
+
+# Elasticsearch configuration
+ELASTICSEARCH_URL=http://localhost:9200
+ELASTICSEARCH_INDEX_PREFIX=labrinth
+ELASTICSEARCH_USERNAME=
+ELASTICSEARCH_PASSWORD=
 
 REDIS_URL=redis://localhost
 REDIS_MIN_CONNECTIONS=0

apps/labrinth/Cargo.toml

@@ -25,7 +25,7 @@ async-stripe = { workspace = true, features = [
   "billing",
   "checkout",
   "connect",
-  "webhook-events",
+  "webhook-events"
 ] }
 async-trait = { workspace = true }
 base64 = { workspace = true }
@@ -43,6 +43,7 @@ deadpool-redis.workspace = true
 derive_more = { workspace = true, features = ["deref", "deref_mut"] }
 dotenvy = { workspace = true }
 either = { workspace = true }
+elasticsearch = { workspace = true, features = ["experimental-apis"] }
 eyre = { workspace = true }
 futures = { workspace = true }
 futures-util = { workspace = true }
@@ -85,11 +86,11 @@ reqwest = { workspace = true, features = [
   "http2",
   "json",
   "multipart",
-  "rustls-tls-webpki-roots",
+  "rustls-tls-webpki-roots"
 ] }
 rust_decimal = { workspace = true, features = [
   "serde-with-float",
-  "serde-with-str",
+  "serde-with-str"
 ] }
 rust_iso3166 = { workspace = true }
 rust-s3 = { workspace = true }

apps/labrinth/src/auth/mod.rs

@@ -20,8 +20,6 @@ use thiserror::Error;
 pub enum AuthenticationError {
     #[error(transparent)]
     Internal(#[from] eyre::Report),
-    #[error("Environment Error")]
-    Env(#[from] dotenvy::Error),
     #[error("An unknown database error occurred: {0}")]
     Sqlx(#[from] sqlx::Error),
     #[error("Database Error: {0}")]
@@ -58,7 +56,6 @@ impl actix_web::ResponseError for AuthenticationError {
             AuthenticationError::Internal(..) => {
                 StatusCode::INTERNAL_SERVER_ERROR
             }
-            AuthenticationError::Env(..) => StatusCode::INTERNAL_SERVER_ERROR,
             AuthenticationError::Sqlx(..) => StatusCode::INTERNAL_SERVER_ERROR,
             AuthenticationError::Database(..) => {
                 StatusCode::INTERNAL_SERVER_ERROR
@@ -94,7 +91,6 @@ impl AuthenticationError {
     pub fn error_name(&self) -> &'static str {
         match self {
             AuthenticationError::Internal(..) => "internal_error",
-            AuthenticationError::Env(..) => "environment_error",
             AuthenticationError::Sqlx(..) => "database_error",
             AuthenticationError::Database(..) => "database_error",
             AuthenticationError::SerDe(..) => "invalid_input",

apps/labrinth/src/background_task.rs

@@ -1,3 +1,4 @@
+use crate::database;
 use crate::database::PgPool;
 use crate::database::redis::RedisPool;
 use crate::queue::billing::{index_billing, index_subscriptions};
@@ -7,9 +8,9 @@ use crate::queue::payouts::{
     insert_bank_balances_and_webhook, process_affiliate_payouts,
     process_payout, remove_payouts_for_refunded_charges,
 };
-use crate::search::indexing::index_projects;
+use crate::search::SearchBackend;
 use crate::util::anrok;
-use crate::{database, search};
+use actix_web::web;
 use clap::ValueEnum;
 use tracing::{error, info, warn};
 
@@ -34,18 +35,19 @@ impl BackgroundTask {
         pool: PgPool,
         ro_pool: PgPool,
         redis_pool: RedisPool,
-        search_config: search::SearchConfig,
+        search_backend: web::Data<dyn SearchBackend>,
         clickhouse: clickhouse::Client,
         stripe_client: stripe::Client,
         anrok_client: anrok::Client,
         email_queue: EmailQueue,
         mural_client: muralpay::Client,
-    ) {
+    ) -> eyre::Result<()> {
         use BackgroundTask::*;
+        // TODO: all of these tasks should return `eyre::Result`s
         match self {
             Migrations => run_migrations().await,
             IndexSearch => {
-                index_search(ro_pool, redis_pool, search_config).await
+                return index_search(ro_pool, redis_pool, search_backend).await;
             }
             ReleaseScheduled => release_scheduled(pool).await,
             UpdateVersions => update_versions(pool, redis_pool).await,
@@ -77,6 +79,7 @@ impl BackgroundTask {
                 run_email(email_queue).await;
             }
         }
+        Ok(())
     }
 }
 
@@ -122,14 +125,10 @@ pub async fn run_migrations() {
 pub async fn index_search(
     ro_pool: PgPool,
     redis_pool: RedisPool,
-    search_config: search::SearchConfig,
-) {
+    search_backend: web::Data<dyn SearchBackend>,
+) -> eyre::Result<()> {
     info!("Indexing local database");
-    let result = index_projects(ro_pool, redis_pool, &search_config).await;
-    if let Err(e) = result {
-        warn!("Local project indexing failed: {:?}", e);
-    }
-    info!("Done indexing local database");
+    search_backend.index_projects(ro_pool, redis_pool).await
 }
 
 pub async fn release_scheduled(pool: PgPool) {

apps/labrinth/src/env.rs

@@ -82,6 +82,7 @@ where
 }
 
 pub fn init() -> eyre::Result<()> {
+    dotenvy::dotenv().ok();
     EnvVars::from_env()?;
     LazyLock::force(&ENV);
     Ok(())
@@ -128,9 +129,6 @@ vars! {
     LABRINTH_EXTERNAL_NOTIFICATION_KEY: String;
     RATE_LIMIT_IGNORE_KEY: String;
     DATABASE_URL: String;
-    MEILISEARCH_READ_ADDR: String;
-    MEILISEARCH_WRITE_ADDRS: StringCsv;
-    MEILISEARCH_KEY: String;
     REDIS_URL: String;
     BIND_ADDR: String;
     SELF_ADDR: String;
@@ -142,6 +140,16 @@ vars! {
     ALLOWED_CALLBACK_URLS: Json<Vec<String>>;
     ANALYTICS_ALLOWED_ORIGINS: Json<Vec<String>>;
 
+    // search
+    SEARCH_BACKEND: crate::search::SearchBackendKind;
+    MEILISEARCH_READ_ADDR: String;
+    MEILISEARCH_WRITE_ADDRS: StringCsv;
+    MEILISEARCH_KEY: String;
+    ELASTICSEARCH_URL: String;
+    ELASTICSEARCH_INDEX_PREFIX: String;
+    ELASTICSEARCH_USERNAME: String = "";
+    ELASTICSEARCH_PASSWORD: String = "";
+
     // storage
     STORAGE_BACKEND: crate::file_hosting::FileHostKind;