Skip to main content

Module Registration Reference

Complete reference for registering modules with the Constellation runtime, organizing custom standard libraries, and managing large module sets.

Quick Navigation

Registration Overview​

Core Concepts​

Module Registration is the process of making modules available to the Constellation runtime so they can be called from constellation-lang pipelines.

import cats.effect.IO
import cats.implicits._
import io.constellation._

// 1. Create a Constellation instance
val constellation = ConstellationImpl.init

// 2. Register modules
constellation.flatMap { c =>
  c.setModule(myModule)
}

// 3. Use modules in pipelines

Key Registration Interfaces:

  • Constellation.setModule(module: Module.Uninitialized): IO[Unit] - Register a single module
  • ModuleRegistry - Internal registry managing name → module mappings
  • Module.Uninitialized - Module template ready for registration
  • Module.Initialized - Module with runtime context after initialization

Registration Workflow​

┌─────────────────────┐
│ Define Module       │
│ (ModuleBuilder)     │
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│ Build Module        │
│ (.build)            │
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│ Register Module     │
│ (setModule)         │
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│ Use in Pipelines    │
│ (constellation-lang)│
└─────────────────────┘

What Happens During Registration:

  1. Name Validation - Ensures module name is unique
  2. Spec Storage - Stores module spec in registry
  3. Factory Storage - Stores uninitialized module for later initialization
  4. Ready for Use - Module can now be referenced in DAG compilation

Single Module Registration​

Basic Registration​

Register a single module with the runtime:

import cats.effect.IO
import io.constellation._

case class TextInput(text: String)
case class TextOutput(result: String)

val uppercaseModule = ModuleBuilder
  .metadata("Uppercase", "Convert text to uppercase", 1, 0)
  .implementationPure[TextInput, TextOutput] { input =>
    TextOutput(input.text.toUpperCase)
  }
  .build

// Register the module
val program: IO[Unit] = for {
  constellation <- ConstellationImpl.init
  _             <- constellation.setModule(uppercaseModule)
} yield ()

Registering with IO-Based Modules​

For modules that perform side effects:

case class ApiInput(endpoint: String)
case class ApiOutput(data: String)

val apiCallModule = ModuleBuilder
  .metadata("ApiCall", "Make HTTP API call", 1, 0)
  .implementation[ApiInput, ApiOutput] { input =>
    IO {
      // Perform HTTP request
      val response = scala.io.Source.fromURL(input.endpoint).mkString
      ApiOutput(response)
    }
  }
  .build

// Registration is the same
constellation.setModule(apiCallModule)

Registering Modules with Context​

Modules can include definition context metadata:

import io.circe.Json
import io.circe.syntax._

val contextualModule = ModuleBuilder
  .metadata("Contextual", "Module with context", 1, 0)
  .definitionContext(Map(
    "author" -> "John Doe".asJson,
    "source" -> "custom-library".asJson,
    "license" -> "MIT".asJson
  ))
  .implementationPure[TextInput, TextOutput] { input =>
    TextOutput(input.text.toUpperCase)
  }
  .build

constellation.setModule(contextualModule)

Error Handling During Registration​

Registration can fail if module names conflict:

val safeRegistration: IO[Unit] = for {
  constellation <- ConstellationImpl.init

  // Check if module already exists
  existing <- constellation.getModuleByName("Uppercase")

  _ <- existing match {
    case Some(_) =>
      IO.println("Module already registered, skipping")
    case None =>
      constellation.setModule(uppercaseModule)
  }
} yield ()

Batch Registration Patterns​

Using traverse for Multiple Modules​

The idiomatic way to register multiple modules in Cats Effect:

import cats.effect.IO
import cats.implicits._  // Required for .traverse
import io.constellation._

val modules: List[Module.Uninitialized] = List(
  uppercaseModule,
  lowercaseModule,
  trimModule
)

val program: IO[Unit] = for {
  constellation <- ConstellationImpl.init
  _             <- modules.traverse(constellation.setModule)
} yield ()

Why traverse:

  • Transforms List[Module] → List[IO[Unit]] → IO[List[Unit]]
  • Executes registrations sequentially
  • Short-circuits on first error
  • Type-safe and composable

Registration with Error Accumulation​

Continue registering modules even if some fail:

import cats.effect.IO
import cats.implicits._

def registerAllOrReport(
  constellation: Constellation,
  modules: List[Module.Uninitialized]
): IO[List[Either[Throwable, Unit]]] = {
  modules.traverse { module =>
    constellation.setModule(module).attempt
  }
}

// Usage
val results = for {
  c       <- ConstellationImpl.init
  results <- registerAllOrReport(c, modules)
  _       <- IO.println(s"Registered ${results.count(_.isRight)} of ${modules.length} modules")
} yield results

Parallel Registration (Advanced)​

Register independent modules in parallel for faster startup:

import cats.effect.IO
import cats.implicits._

val parallelRegistration: IO[Unit] = for {
  constellation <- ConstellationImpl.init

  // Register modules in parallel
  _ <- modules.parTraverse(constellation.setModule)
} yield ()

⚠️ Caution:

  • Only use if module initialization is expensive and modules are independent
  • Module registry must be thread-safe (default implementation is)
  • Consider using bounded parallelism: parTraverseN(8)(constellation.setModule)

Conditional Registration​

Register modules based on runtime conditions:

import cats.effect.IO

def registerConditionally(
  constellation: Constellation,
  modules: List[Module.Uninitialized],
  enableFeatures: Set[String]
): IO[Unit] = {
  val filtered = modules.filter { module =>
    module.spec.metadata.tags.exists(enableFeatures.contains)
  }

  filtered.traverse(constellation.setModule).void
}

// Usage
val program = for {
  c <- ConstellationImpl.init
  _ <- registerConditionally(
    c,
    allModules,
    Set("text", "math")  // Only register text and math modules
  )
} yield ()

Organizing Modules into Libraries​

Category-Based Organization​

Split modules into logical categories using traits (stdlib pattern):

package io.mycompany.modules

import io.constellation._

// modules/TextFunctions.scala
trait TextFunctions {
  // Module definitions
  val uppercaseModule: Module.Uninitialized = ...
  val lowercaseModule: Module.Uninitialized = ...
  val trimModule: Module.Uninitialized = ...

  // Collect modules into a map
  def textModules: Map[String, Module.Uninitialized] = Map(
    uppercaseModule.spec.name -> uppercaseModule,
    lowercaseModule.spec.name -> lowercaseModule,
    trimModule.spec.name      -> trimModule
  )
}

// modules/MathFunctions.scala
trait MathFunctions {
  val addModule: Module.Uninitialized = ...
  val subtractModule: Module.Uninitialized = ...

  def mathModules: Map[String, Module.Uninitialized] = Map(
    addModule.spec.name      -> addModule,
    subtractModule.spec.name -> subtractModule
  )
}

// MyLib.scala
object MyLib extends TextFunctions with MathFunctions {
  def allModules: Map[String, Module.Uninitialized] =
    textModules ++ mathModules
}

Benefits:

  • Clear separation of concerns
  • Easy to add/remove categories
  • Each category can be tested independently
  • IDE-friendly organization

Object-Based Organization​

Group related modules in companion objects:

package io.mycompany.modules

import io.constellation._

object TextModules {
  case class TextInput(text: String)
  case class TextOutput(result: String)

  val uppercase: Module.Uninitialized = ModuleBuilder
    .metadata("Uppercase", "Convert to uppercase", 1, 0)
    .implementationPure[TextInput, TextOutput](in => TextOutput(in.text.toUpperCase))
    .build

  val lowercase: Module.Uninitialized = ModuleBuilder
    .metadata("Lowercase", "Convert to lowercase", 1, 0)
    .implementationPure[TextInput, TextOutput](in => TextOutput(in.text.toLowerCase))
    .build

  val all: List[Module.Uninitialized] = List(uppercase, lowercase)
}

object DataModules {
  val sumList: Module.Uninitialized = ...
  val average: Module.Uninitialized = ...

  val all: List[Module.Uninitialized] = List(sumList, average)
}

// Usage
val allModules = TextModules.all ++ DataModules.all

Multi-Package Organization​

For large libraries, split into packages:

src/main/scala/io/mycompany/
├── MyLib.scala                    (top-level facade)
└── modules/
    ├── text/
    │   ├── TextModules.scala
    │   └── TextSignatures.scala
    ├── data/
    │   ├── DataModules.scala
    │   └── DataSignatures.scala
    └── network/
        ├── HttpModules.scala
        └── HttpSignatures.scala

MyLib.scala:

package io.mycompany

import io.constellation._
import io.mycompany.modules.text.TextModules
import io.mycompany.modules.data.DataModules
import io.mycompany.modules.network.HttpModules

object MyLib {
  def allModules: Map[String, Module.Uninitialized] =
    TextModules.all ++ DataModules.all ++ HttpModules.all

  def registerAll(constellation: Constellation): IO[Unit] =
    allModules.values.toList.traverse(constellation.setModule).void
}

Custom Stdlib Patterns​

Creating a Domain-Specific Standard Library​

Build a custom stdlib for a specific domain (e.g., e-commerce, finance, ML):

package io.mycompany.ecommerce

import cats.effect.IO
import cats.implicits._
import io.constellation._
import io.constellation.lang.LangCompilerBuilder
import io.constellation.lang.semantic._
import io.constellation.stdlib.StdLib

// 1. Define modules
trait ProductModules {
  case class ProductInput(sku: String)
  case class ProductOutput(name: String, price: Double)

  val getProduct: Module.Uninitialized = ModuleBuilder
    .metadata("GetProduct", "Fetch product by SKU", 1, 0)
    .tags("ecommerce", "product")
    .implementation[ProductInput, ProductOutput] { input =>
      IO {
        // Database lookup
        ProductOutput("Widget", 29.99)
      }
    }
    .build

  def productModules: Map[String, Module.Uninitialized] = Map(
    getProduct.spec.name -> getProduct
  )
}

trait OrderModules {
  case class OrderInput(userId: String, items: List[String])
  case class OrderOutput(orderId: String, total: Double)

  val createOrder: Module.Uninitialized = ModuleBuilder
    .metadata("CreateOrder", "Create new order", 1, 0)
    .tags("ecommerce", "order")
    .implementation[OrderInput, OrderOutput] { input =>
      IO {
        OrderOutput("ORD-123", 99.99)
      }
    }
    .build

  def orderModules: Map[String, Module.Uninitialized] = Map(
    createOrder.spec.name -> createOrder
  )
}

// 2. Define function signatures for type checking
trait EcommerceSignatures {
  val getProductSig = FunctionSignature(
    name = "GetProduct",
    params = List("sku" -> SemanticType.SString),
    returns = SemanticType.SRecord(Map(
      "name" -> SemanticType.SString,
      "price" -> SemanticType.SFloat
    )),
    moduleName = "GetProduct"
  )

  val createOrderSig = FunctionSignature(
    name = "CreateOrder",
    params = List(
      "userId" -> SemanticType.SString,
      "items" -> SemanticType.SList(SemanticType.SString)
    ),
    returns = SemanticType.SRecord(Map(
      "orderId" -> SemanticType.SString,
      "total" -> SemanticType.SFloat
    )),
    moduleName = "CreateOrder"
  )

  def ecommerceSignatures: List[FunctionSignature] = List(
    getProductSig,
    createOrderSig
  )
}

// 3. Combine into a custom stdlib
object EcommerceStdLib
  extends ProductModules
  with OrderModules
  with EcommerceSignatures {

  def allModules: Map[String, Module.Uninitialized] =
    productModules ++ orderModules

  def allSignatures: List[FunctionSignature] =
    ecommerceSignatures

  // Register with compiler builder
  def registerAll(builder: LangCompilerBuilder): LangCompilerBuilder =
    allSignatures.foldLeft(builder)((b, sig) => b.withFunction(sig))

  // Create a compiler with both base stdlib and ecommerce stdlib
  def compiler: LangCompiler = {
    val combinedModules = StdLib.allModules ++ allModules
    val builder = registerAll(StdLib.registerAll(LangCompilerBuilder()))
      .withModules(combinedModules)
    builder.build
  }
}

Usage:

val program = for {
  // Use combined compiler
  compiler <- IO.pure(EcommerceStdLib.compiler)

  // Compile pipeline using both base and ecommerce functions
  result <- compiler.compile("""
    in sku: String

    product = GetProduct(sku)

    # Can also use base stdlib functions
    formatted = concat("Product: ", product.name)

    out formatted
  """, "product-lookup")
} yield result

Extending Existing Stdlibs​

Add domain modules while preserving the base stdlib:

object ExtendedStdLib {
  // Your domain modules
  val customModules: Map[String, Module.Uninitialized] = Map(
    "CustomModule" -> myCustomModule
  )

  // Combine with base stdlib
  def allModules: Map[String, Module.Uninitialized] =
    StdLib.allModules ++ customModules

  // Register all modules with Constellation
  def registerAll(constellation: Constellation): IO[Unit] =
    allModules.values.toList.traverse(constellation.setModule).void
}

Plugin-Based Architecture​

Support loading modules from plugins:

trait ModulePlugin {
  def name: String
  def modules: Map[String, Module.Uninitialized]
  def signatures: List[FunctionSignature]
}

class PluggableStdLib(plugins: List[ModulePlugin]) {
  def allModules: Map[String, Module.Uninitialized] =
    plugins.flatMap(_.modules).toMap

  def allSignatures: List[FunctionSignature] =
    plugins.flatMap(_.signatures)

  def registerAll(builder: LangCompilerBuilder): LangCompilerBuilder =
    allSignatures.foldLeft(builder)((b, sig) => b.withFunction(sig))
}

// Plugin implementation
object TextProcessingPlugin extends ModulePlugin {
  def name = "text-processing"

  def modules = Map(
    "Uppercase" -> uppercaseModule,
    "Lowercase" -> lowercaseModule
  )

  def signatures = List(uppercaseSig, lowercaseSig)
}

// Usage
val stdLib = new PluggableStdLib(List(
  TextProcessingPlugin,
  DataProcessingPlugin,
  NetworkPlugin
))

Dynamic Module Loading​

Loading Modules at Runtime​

Load and register modules based on configuration:

import cats.effect.IO
import io.circe.parser._

case class ModuleConfig(
  name: String,
  enabled: Boolean,
  tags: List[String]
)

def loadModulesFromConfig(configJson: String): IO[List[Module.Uninitialized]] = {
  for {
    config <- IO.fromEither(decode[List[ModuleConfig]](configJson))

    // Map config to actual module instances
    modules = config.filter(_.enabled).flatMap { cfg =>
      ModuleRegistry.lookupByName(cfg.name)
    }
  } yield modules
}

val program = for {
  config      <- IO(scala.io.Source.fromFile("modules.json").mkString)
  modules     <- loadModulesFromConfig(config)
  constellation <- ConstellationImpl.init
  _           <- modules.traverse(constellation.setModule)
} yield ()

Lazy Module Registration​

Register modules on-demand when first referenced:

import cats.effect.Ref

class LazyModuleRegistry(
  constellation: Constellation,
  moduleFactory: String => Option[Module.Uninitialized]
) {
  private val registered = Ref.unsafe[IO, Set[String]](Set.empty)

  def ensureRegistered(moduleName: String): IO[Unit] = {
    registered.get.flatMap { reg =>
      if (reg.contains(moduleName)) {
        IO.unit
      } else {
        moduleFactory(moduleName) match {
          case Some(module) =>
            for {
              _ <- constellation.setModule(module)
              _ <- registered.update(_ + moduleName)
            } yield ()
          case None =>
            IO.raiseError(new Exception(s"Unknown module: $moduleName"))
        }
      }
    }
  }
}

Hot Module Reloading​

Support updating modules without restarting:

class ReloadableModuleRegistry(constellation: Constellation) {
  def reloadModule(
    name: String,
    newModule: Module.Uninitialized
  ): IO[Unit] = {
    // Note: Constellation doesn't support un-registration
    // so this replaces the module implementation
    constellation.setModule(newModule)
  }

  def reloadFromSource(
    name: String,
    sourceCode: String
  ): IO[Unit] = {
    for {
      // Compile new module from source
      module <- compileModuleSource(sourceCode)

      // Register (replaces existing)
      _ <- constellation.setModule(module)

      _ <- IO.println(s"Reloaded module: $name")
    } yield ()
  }
}

Module Versioning and Updates​

Semantic Versioning​

Use semantic versioning in module metadata:

val moduleV1 = ModuleBuilder
  .metadata("MyModule", "Description", majorVersion = 1, minorVersion = 0)
  .implementationPure[Input, Output] { input => ... }
  .build

val moduleV2 = ModuleBuilder
  .metadata("MyModule", "Description", majorVersion = 2, minorVersion = 0)
  .implementationPure[Input, Output] { input =>
    // Breaking change - new implementation
    ...
  }
  .build

Side-by-Side Versioning​

Support multiple versions simultaneously with naming:

val moduleV1 = ModuleBuilder
  .metadata("MyModule_v1", "Description v1", 1, 0)
  .implementationPure[InputV1, OutputV1] { ... }
  .build

val moduleV2 = ModuleBuilder
  .metadata("MyModule_v2", "Description v2", 2, 0)
  .implementationPure[InputV2, OutputV2] { ... }
  .build

// Register both
constellation.setModule(moduleV1)
constellation.setModule(moduleV2)

In constellation-lang:

# Use specific version
result_v1 = MyModule_v1(input)
result_v2 = MyModule_v2(input)

Version Aliasing​

Point "latest" alias to current version:

object ModuleVersions {
  val v1: Module.Uninitialized = ...
  val v2: Module.Uninitialized = ...

  // Current production version
  val latest: Module.Uninitialized = v2

  def registerAll(constellation: Constellation): IO[Unit] = {
    List(
      v1,
      v2,
      latest.spec.copy(metadata = latest.spec.metadata.copy(name = "MyModule"))
    ).traverse(constellation.setModule).void
  }
}

Deprecation Strategy​

Mark deprecated modules with tags and context:

val deprecatedModule = ModuleBuilder
  .metadata("OldModule", "DEPRECATED: Use NewModule instead", 1, 0)
  .tags("deprecated", "legacy")
  .definitionContext(Map(
    "deprecated" -> true.asJson,
    "deprecatedSince" -> "2024-01-01".asJson,
    "replacement" -> "NewModule".asJson
  ))
  .implementationPure[Input, Output] { ... }
  .build

Namespace Management​

Prefixed Naming​

Use dot-notation for namespacing:

trait MathModules {
  val add = ModuleBuilder
    .metadata("stdlib.math.add", "Add two integers", 1, 0)
    .tags("stdlib", "math")
    .implementationPure[TwoInts, IntOut] { ... }
    .build

  val multiply = ModuleBuilder
    .metadata("stdlib.math.multiply", "Multiply two integers", 1, 0)
    .tags("stdlib", "math")
    .implementationPure[TwoInts, IntOut] { ... }
    .build
}

trait StringModules {
  val concat = ModuleBuilder
    .metadata("stdlib.string.concat", "Concatenate strings", 1, 0)
    .tags("stdlib", "string")
    .implementationPure[TwoStrings, StringOut] { ... }
    .build
}

Benefits:

  • Clear ownership and category
  • Prevents name collisions
  • IDE autocomplete friendly
  • Easy to filter by prefix

Organization-Based Namespacing​

Prefix modules with organization or team name:

// Team-specific modules
val orderModule = ModuleBuilder
  .metadata("acme.sales.CreateOrder", "Create sales order", 1, 0)
  .tags("acme", "sales")
  .implementationPure[OrderInput, OrderOutput] { ... }
  .build

val inventoryModule = ModuleBuilder
  .metadata("acme.inventory.CheckStock", "Check inventory", 1, 0)
  .tags("acme", "inventory")
  .implementationPure[StockInput, StockOutput] { ... }
  .build

Namespace Helpers​

Build utilities for consistent naming:

object NamespaceHelper {
  def namespaced(category: String, name: String): String =
    s"mycompany.$category.$name"

  def buildModule[I <: Product, O <: Product](
    category: String,
    name: String,
    description: String
  )(impl: I => O): ModuleBuilderInit = {
    ModuleBuilder.metadata(
      name = namespaced(category, name),
      description = description,
      majorVersion = 1,
      minorVersion = 0
    ).tags("mycompany", category)
  }
}

// Usage
val uppercase = NamespaceHelper
  .buildModule[TextInput, TextOutput]("text", "Uppercase", "Convert to uppercase")
  .implementationPure[TextInput, TextOutput](in => TextOutput(in.text.toUpperCase))
  .build
// Results in module named: "mycompany.text.Uppercase"

Namespace Filtering​

Register only modules matching a namespace:

def registerNamespace(
  constellation: Constellation,
  modules: Map[String, Module.Uninitialized],
  namespace: String
): IO[Unit] = {
  val filtered = modules.filter { case (name, _) =>
    name.startsWith(s"$namespace.")
  }
  filtered.values.toList.traverse(constellation.setModule).void
}

// Register only "stdlib.math.*" modules
registerNamespace(constellation, allModules, "stdlib.math")

Module Discovery and Introspection​

Listing Registered Modules​

Query all registered modules:

val program = for {
  constellation <- ConstellationImpl.init
  _             <- allModules.traverse(constellation.setModule)

  // List all registered modules
  specs <- constellation.getModules

  _ <- IO.println(s"Registered ${specs.length} modules:")
  _ <- specs.traverse { spec =>
    IO.println(s"  - ${spec.name}: ${spec.description}")
  }
} yield ()

Module Lookup by Name​

Check if a specific module is registered:

val lookupModule = for {
  constellation <- ConstellationImpl.init

  // Try to get a specific module
  maybeModule <- constellation.getModuleByName("Uppercase")

  _ <- maybeModule match {
    case Some(module) =>
      IO.println(s"Found: ${module.spec.metadata.name}")
    case None =>
      IO.println("Module not found")
  }
} yield ()

Query Modules by Tags​

Filter modules by tags:

def findModulesByTag(
  constellation: Constellation,
  tag: String
): IO[List[ModuleNodeSpec]] = {
  constellation.getModules.map { specs =>
    specs.filter(_.metadata.tags.contains(tag))
  }
}

// Usage
val textModules = for {
  c <- ConstellationImpl.init
  _ <- registerAll(c)
  modules <- findModulesByTag(c, "text")
  _ <- IO.println(s"Found ${modules.length} text modules")
} yield modules

Build Module Directory​

Generate documentation from registered modules:

case class ModuleInfo(
  name: String,
  description: String,
  version: String,
  tags: List[String]
)

def buildModuleDirectory(
  constellation: Constellation
): IO[List[ModuleInfo]] = {
  constellation.getModules.map { specs =>
    specs.map { spec =>
      ModuleInfo(
        name = spec.metadata.name,
        description = spec.metadata.description,
        version = s"${spec.metadata.majorVersion}.${spec.metadata.minorVersion}",
        tags = spec.metadata.tags
      )
    }
  }
}

// Generate JSON catalog
val catalog = for {
  c <- ConstellationImpl.init
  _ <- registerAll(c)
  dir <- buildModuleDirectory(c)
  json = dir.asJson.spaces2
  _ <- IO(scala.io.File("module-catalog.json").write(json))
} yield ()

Health Check Modules​

Verify all required modules are registered:

def checkRequiredModules(
  constellation: Constellation,
  required: Set[String]
): IO[Either[List[String], Unit]] = {
  for {
    specs <- constellation.getModules
    registered = specs.map(_.metadata.name).toSet
    missing = required -- registered
  } yield {
    if (missing.isEmpty) Right(())
    else Left(missing.toList)
  }
}

// Usage
val healthCheck = for {
  c <- ConstellationImpl.init
  _ <- registerAll(c)

  result <- checkRequiredModules(c, Set(
    "Uppercase",
    "Lowercase",
    "GetProduct"
  ))

  _ <- result match {
    case Right(_) =>
      IO.println("✓ All required modules registered")
    case Left(missing) =>
      IO.raiseError(new Exception(s"Missing modules: ${missing.mkString(", ")}"))
  }
} yield ()

Best Practices​

Registration at Startup​

Register all modules during application initialization:

object MyApplication extends IOApp {
  def run(args: List[String]): IO[ExitCode] = {
    for {
      // Initialize Constellation
      constellation <- ConstellationImpl.init

      // Register all modules upfront
      _ <- MyLib.registerAll(constellation)

      // Verify registration
      specs <- constellation.getModules
      _ <- IO.println(s"Registered ${specs.length} modules")

      // Start server
      _ <- ConstellationServer.builder(constellation, compiler).run
    } yield ExitCode.Success
  }
}

Module Organization Checklist​

For production deployments:

✅ Group by Category - Use traits or objects to organize by domain ✅ Use Namespacing - Prefix module names with category/org ✅ Version Explicitly - Include version in metadata ✅ Tag Appropriately - Add tags for filtering and discovery ✅ Document Thoroughly - Write clear descriptions ✅ Test Registration - Verify all modules register successfully ✅ Health Checks - Check required modules at startup

Performance Considerations​

Registration Performance:

  • Registration is fast (metadata storage only)
  • No need to optimize unless registering 1000s of modules
  • Consider lazy registration only if module initialization is expensive

Best Practices:

// ✅ Good: Register all at startup
modules.traverse(constellation.setModule)

// ❌ Avoid: Registering on every request
def handleRequest(moduleName: String) = {
  constellation.setModule(getModule(moduleName))  // Don't do this
}

// ✅ Good: Register once, use many times
val app = for {
  c <- ConstellationImpl.init
  _ <- modules.traverse(c.setModule)  // Once at startup

  // Use registered modules in requests
  _ <- handleRequests(c)
} yield ()

Error Handling Best Practices​

Handle registration errors gracefully:

def registerWithErrorHandling(
  constellation: Constellation,
  modules: List[Module.Uninitialized]
): IO[Unit] = {
  modules.zipWithIndex.traverse { case (module, idx) =>
    constellation.setModule(module).handleErrorWith { error =>
      IO.println(s"Failed to register module ${idx + 1}: ${module.spec.metadata.name}") *>
      IO.println(s"  Error: ${error.getMessage}") *>
      IO.raiseError(error)  // Re-raise to fail fast
    }
  }.void
}

Testing Module Registration​

Unit test your module registration:

import munit.CatsEffectSuite

class ModuleRegistrationTest extends CatsEffectSuite {
  test("all modules register successfully") {
    for {
      constellation <- ConstellationImpl.init
      _             <- MyLib.allModules.values.toList.traverse(constellation.setModule)
      specs         <- constellation.getModules

      // Verify count
      _ = assertEquals(specs.length, MyLib.allModules.size)

      // Verify all present
      names = specs.map(_.metadata.name).toSet
      _ = assert(names.contains("Uppercase"))
      _ = assert(names.contains("Lowercase"))
    } yield ()
  }

  test("can retrieve registered module by name") {
    for {
      constellation <- ConstellationImpl.init
      _             <- constellation.setModule(MyLib.uppercaseModule)
      maybeModule   <- constellation.getModuleByName("Uppercase")

      _ = assert(maybeModule.isDefined)
      _ = assertEquals(maybeModule.get.spec.metadata.name, "Uppercase")
    } yield ()
  }
}

Monitoring Module Usage​

Track which modules are actually used:

class MonitoredModuleRegistry(constellation: Constellation) {
  private val usageCounts = Ref.unsafe[IO, Map[String, Long]](Map.empty)

  def recordUsage(moduleName: String): IO[Unit] =
    usageCounts.update { counts =>
      counts.updated(moduleName, counts.getOrElse(moduleName, 0L) + 1)
    }

  def getUsageStats: IO[Map[String, Long]] =
    usageCounts.get

  def getUnusedModules: IO[List[String]] = {
    for {
      registered <- constellation.getModules
      usage      <- usageCounts.get

      unused = registered
        .map(_.metadata.name)
        .filterNot(usage.contains)
    } yield unused
  }
}

Module Registry Patterns Summary​

PatternUse CaseProsCons
Single RegistrationSmall apps, single moduleSimple, directDoesn't scale
Batch with traverseMedium apps (10-100 modules)Type-safe, composableSequential
Parallel RegistrationLarge apps (100+ modules)Fast startupRequires thread safety
Lazy LoadingOn-demand modulesMemory efficientComplex, race conditions
Plugin ArchitectureExtensible systemsHighly modularRequires plugin API
Namespace PrefixingLarge teamsPrevents collisionsVerbose names
Version AliasingMulti-version supportGradual migrationComplexity

Complete Examples​

Minimal Example​

import cats.effect.{IO, IOApp, ExitCode}
import cats.implicits._
import io.constellation._

object MinimalApp extends IOApp {
  case class TextInput(text: String)
  case class TextOutput(result: String)

  val uppercaseModule = ModuleBuilder
    .metadata("Uppercase", "Convert to uppercase", 1, 0)
    .implementationPure[TextInput, TextOutput](in => TextOutput(in.text.toUpperCase))
    .build

  def run(args: List[String]): IO[ExitCode] = {
    for {
      constellation <- ConstellationImpl.init
      _             <- constellation.setModule(uppercaseModule)
      specs         <- constellation.getModules
      _             <- IO.println(s"Registered ${specs.length} module(s)")
    } yield ExitCode.Success
  }
}

Production Example​

import cats.effect.{IO, IOApp, ExitCode, Resource}
import cats.implicits._
import io.constellation._
import io.constellation.lang.LangCompilerBuilder
import io.constellation.stdlib.StdLib

// 1. Define custom modules
object MyCompanyModules {
  // Product modules
  val getProduct = ModuleBuilder
    .metadata("myco.product.Get", "Fetch product by SKU", 1, 0)
    .tags("myco", "product")
    .implementation[ProductInput, ProductOutput] { ... }
    .build

  // Order modules
  val createOrder = ModuleBuilder
    .metadata("myco.order.Create", "Create new order", 1, 0)
    .tags("myco", "order")
    .implementation[OrderInput, OrderOutput] { ... }
    .build

  val all: List[Module.Uninitialized] = List(
    getProduct,
    createOrder
  )
}

// 2. Application setup
object MyApplication extends IOApp {
  def setupConstellation: IO[Constellation] = {
    for {
      // Initialize
      constellation <- ConstellationImpl.init

      // Register stdlib
      _ <- StdLib.allModules.values.toList.traverse(constellation.setModule)

      // Register custom modules
      _ <- MyCompanyModules.all.traverse(constellation.setModule)

      // Verify registration
      specs <- constellation.getModules
      _ <- IO.println(s"✓ Registered ${specs.length} modules")

      // Health check
      _ <- checkRequiredModules(constellation, Set(
        "myco.product.Get",
        "myco.order.Create"
      )).flatMap {
        case Right(_) => IO.println("✓ All required modules present")
        case Left(missing) => IO.raiseError(new Exception(s"Missing: ${missing.mkString(", ")}"))
      }
    } yield constellation
  }

  def checkRequiredModules(
    constellation: Constellation,
    required: Set[String]
  ): IO[Either[List[String], Unit]] = {
    constellation.getModules.map { specs =>
      val registered = specs.map(_.metadata.name).toSet
      val missing = required -- registered
      if (missing.isEmpty) Right(()) else Left(missing.toList)
    }
  }

  def run(args: List[String]): IO[ExitCode] = {
    setupConstellation.flatMap { constellation =>
      // Run server or application logic
      IO.println("Application ready") *> IO.pure(ExitCode.Success)
    }
  }
}

Multi-Environment Example​

object EnvironmentAwareRegistry {
  sealed trait Environment
  case object Development extends Environment
  case object Staging extends Environment
  case object Production extends Environment

  def getEnvironment: IO[Environment] = {
    IO(System.getenv("APP_ENV")).map {
      case "production" => Production
      case "staging" => Staging
      case _ => Development
    }
  }

  def modulesForEnvironment(env: Environment): List[Module.Uninitialized] = {
    val baseModules = StdLib.allModules.values.toList

    val envSpecific = env match {
      case Development =>
        // Include debug modules in dev
        baseModules ++ DebugModules.all

      case Staging =>
        // Include monitoring in staging
        baseModules ++ MonitoringModules.all

      case Production =>
        // Production-optimized modules only
        baseModules.filterNot(_.spec.metadata.tags.contains("debug"))
    }

    envSpecific
  }

  def setup: IO[Constellation] = {
    for {
      env           <- getEnvironment
      modules       = modulesForEnvironment(env)
      constellation <- ConstellationImpl.init
      _             <- modules.traverse(constellation.setModule)
      _             <- IO.println(s"Registered ${modules.length} modules for $env")
    } yield constellation
  }
}

Troubleshooting​

Common Issues​

Problem: Module not found in pipeline execution

Error: Unknown module 'MyModule'

Solution: Verify module is registered before compilation:

for {
  c <- ConstellationImpl.init
  _ <- c.setModule(myModule)  // Must happen before compile

  // Now compile pipeline
  result <- compiler.compile(source, "my-dag")
} yield result

Problem: Name collision during registration

Error: Module 'Uppercase' already registered

Solution: Use unique names or namespacing:

// Option 1: Rename module
val uppercaseV2 = ModuleBuilder
  .metadata("UppercaseV2", "Description", 2, 0)
  .implementationPure[In, Out] { ... }
  .build

// Option 2: Use namespace
val uppercaseText = ModuleBuilder
  .metadata("text.Uppercase", "Description", 1, 0)
  .implementationPure[In, Out] { ... }
  .build

Problem: Module registered but types don't match

Error: Type mismatch for module 'MyModule'

Solution: Ensure FunctionSignature matches ModuleBuilder types:

// Module
case class MyInput(x: Long)
case class MyOutput(y: Long)
val module = ModuleBuilder
  .metadata("MyModule", "Description", 1, 0)
  .implementationPure[MyInput, MyOutput] { ... }
  .build

// Signature MUST match
val signature = FunctionSignature(
  name = "MyModule",
  params = List("x" -> SemanticType.SInt),  // Must match MyInput.x
  returns = SemanticType.SInt,               // Must match MyOutput.y
  moduleName = "MyModule"
)

Problem: Forgot to import cats.implicits._

Error: value traverse is not a member of List[Module.Uninitialized]

Solution: Add the import:

import cats.implicits._  // Required for .traverse

modules.traverse(constellation.setModule)

Cross-Process Modules​

All patterns above register in-process modules that run in the same JVM. For modules that need their own process, language runtime, or independent scaling, see the Module Provider Protocol:

  • External services register modules via gRPC instead of setModule
  • Modules can be written in any language (Python, Go, Rust, etc.)
  • Server handles namespace isolation, heartbeats, and load balancing
  • Trade-off: higher latency (network round-trip) for better isolation and scalability

See: Module Provider Integration

See Also​