Skip to content
Yulin AWS simulator logoYulin AWS Simulator
Development blog

Simulated Glue

Yulin simulates Glue Data Catalog databases, tables and partitions. The catalog stores metadata. It does not read the data in S3.

Simulated Athena reads the catalog and evaluates partition projection when it runs a query.

Glue-specific types are imported from the @kensio/yulin/glue subpath.

Deploying a database and a table

Section titled “Deploying a database and a table”

Simulated CloudFormation deploys AWS::Glue::Database and AWS::Glue::Table. A table can name its database through Ref.

/**
 * Deploying a Glue database and a table, then reading the table back.
 */


import { GetTableCommand } from "@aws-sdk/client-glue";


import { SimAws } from "@kensio/yulin";


const simAws = new SimAws();


const stack = await simAws.cloudFormation().deployTemplate({
  stackName: "analytics-stack",
  template: {
    Resources: {
      LogDatabase: {
        Type: "AWS::Glue::Database",
        Properties: {
          CatalogId: { Ref: "AWS::AccountId" },
          DatabaseInput: { Name: "site_logs" },
        },
      },
      LogTable: {
        Type: "AWS::Glue::Table",
        Properties: {
          CatalogId: { Ref: "AWS::AccountId" },
          DatabaseName: { Ref: "LogDatabase" },
          TableInput: {
            Name: "access_logs",
            TableType: "EXTERNAL_TABLE",
            PartitionKeys: [{ Name: "year", Type: "string" }],
            StorageDescriptor: {
              Columns: [{ Name: "status", Type: "int" }],
              Location: "s3://site-logs/cloudfront/",
            },
            Parameters: {
              "projection.enabled": "true",
              "projection.year.type": "date",
              "projection.year.format": "yyyy",
              "projection.year.range": "2026,NOW",
              // eslint-disable-next-line no-template-curly-in-string
              "storage.location.template": "s3://site-logs/cloudfront/${year}/",
            },
          },
        },
      },
    },
  },
});


await stack.waitForDeployComplete();


const { Table } = simAws
  .glue()
  .getTable(
    new GetTableCommand({ DatabaseName: "site_logs", Name: "access_logs" }),
  );


// true
console.log(Table.Parameters["projection.enabled"]);

Partition projection configuration is stored in TableInput.Parameters. Athena reads these parameters when a query runs and rejects invalid projection configuration.

Ref answers with the database name and with the table name.

Names are folded to lower case

Section titled “Names are folded to lower case”

Database and table names are stored in lower case. A database created as Rainlytics is reported as rainlytics.

/**
 * A catalog name folded on its way in.
 */


import {
  CreateDatabaseCommand,
  GetDatabaseCommand,
} from "@aws-sdk/client-glue";


import { SimAws } from "@kensio/yulin";


const glue = new SimAws().glue();


glue.createDatabase(
  new CreateDatabaseCommand({ DatabaseInput: { Name: "Rainlytics" } }),
);


const { Database } = glue.getDatabase(
  new GetDatabaseCommand({ Name: "Rainlytics" }),
);


// rainlytics
console.log(Database.Name);

Names that differ only by case refer to the same database or table. Creating the second one raises AlreadyExistsException.

Column and partition key names keep their original case.

CloudFormation generates a lower-case name when the template omits one. See generated resource names.

Reading the catalog back

Section titled “Reading the catalog back”

Use the SDK commands to read the catalog through IAM authorization. The findDatabase, findTable, allDatabases, tablesInDatabase, findPartition and partitionsInTable accessors read the same state directly.

Storage descriptor columns and partition keys keep their declared order. Athena rejects a table that repeats a partition key in the storage descriptor columns.

Registering partitions

Section titled “Registering partitions”

Register partitions with the Glue partition commands. Values follow the order of the table’s PartitionKeys. Each partition may provide its own storage descriptor.

/**
 * Registering two days of partitions against a table, then listing them.
 */


import {
  BatchCreatePartitionCommand,
  CreateDatabaseCommand,
  CreateTableCommand,
  GetPartitionsCommand,
} from "@aws-sdk/client-glue";


import { SimAws } from "@kensio/yulin";


const simAws = new SimAws();
const glue = simAws.glue();


glue.createDatabase(
  new CreateDatabaseCommand({ DatabaseInput: { Name: "site_logs" } }),
);
glue.createTable(
  new CreateTableCommand({
    DatabaseName: "site_logs",
    TableInput: {
      Name: "access_logs",
      PartitionKeys: [{ Name: "day", Type: "string" }],
      StorageDescriptor: { Location: "s3://site-logs/cloudfront/" },
    },
  }),
);


const { Errors } = glue.batchCreatePartition(
  new BatchCreatePartitionCommand({
    DatabaseName: "site_logs",
    TableName: "access_logs",
    PartitionInputList: [
      {
        Values: ["2026-08-25"],
        StorageDescriptor: {
          Location: "s3://site-logs/cloudfront/day=2026-08-25/",
        },
      },
      {
        Values: ["2026-08-26"],
        StorageDescriptor: {
          Location: "s3://site-logs/cloudfront/day=2026-08-26/",
        },
      },
    ],
  }),
);


const { Partitions } = glue.getPartitions(
  new GetPartitionsCommand({
    DatabaseName: "site_logs",
    TableName: "access_logs",
  }),
);


// 0
console.log(Errors.length);
// s3://site-logs/cloudfront/day=2026-08-26/
console.log(Partitions[1]?.StorageDescriptor?.Location);

CreatePartition and BatchCreatePartition register partitions. GetPartition reads one by its values. GetPartitions returns them in registration order. The delete commands remove them.

A batch continues after an individual partition fails. Its Errors list contains the supplied values and an ErrorCode for each failure.

The number of values must match the number of partition keys. A mismatch raises InvalidInputException. Registering the same values twice raises AlreadyExistsException. Deleting a table removes its partitions, and deleting a database removes its tables.

Filtering partitions

Section titled “Filtering partitions”

Pass Expression to GetPartitions to filter the result. Omitting it returns every partition.

/**
 * Reading back the partitions an expression matches.
 */


import {
  BatchCreatePartitionCommand,
  CreateDatabaseCommand,
  CreateTableCommand,
  GetPartitionsCommand,
} from "@aws-sdk/client-glue";


import { SimAws } from "@kensio/yulin";


const simAws = new SimAws();
const glue = simAws.glue();


glue.createDatabase(
  new CreateDatabaseCommand({ DatabaseInput: { Name: "site_logs" } }),
);
glue.createTable(
  new CreateTableCommand({
    DatabaseName: "site_logs",
    TableInput: {
      Name: "access_logs",
      PartitionKeys: [
        { Name: "day", Type: "string" },
        { Name: "region", Type: "string" },
      ],
    },
  }),
);


glue.batchCreatePartition(
  new BatchCreatePartitionCommand({
    DatabaseName: "site_logs",
    TableName: "access_logs",
    PartitionInputList: [
      { Values: ["2026-07-31", "eu-west-2"] },
      { Values: ["2026-08-01", "eu-west-2"] },
      { Values: ["2026-08-02", "us-east-1"] },
    ],
  }),
);


const { Partitions } = glue.getPartitions(
  new GetPartitionsCommand({
    DatabaseName: "site_logs",
    TableName: "access_logs",
    Expression: "day >= '2026-08-01' AND region IN ('eu-west-2', 'us-east-1')",
  }),
);


// [["2026-08-01","eu-west-2"],["2026-08-02","us-east-1"]]
console.log(JSON.stringify(Partitions.map((partition) => partition.Values)));

Expressions support =, <>, !=, >, <, >=, <=, LIKE, IN and BETWEEN. Combine terms with AND, OR, NOT and parentheses. Operator precedence is NOT, then AND, then OR.

Numeric partition key types use numeric comparison. Other types use text comparison. A numeric key requires a numeric literal.

LIKE treats the value as text. % matches any sequence and _ matches one character.

An unknown partition key or invalid expression is rejected with a position-aware error.

Intercepting a GlueClient

Section titled “Intercepting a GlueClient”

Intercept a GlueClient when application code constructs the client itself.

/**
 * Ordinary Glue SDK code reaching the simulated catalog.
 */


import {
  CreateDatabaseCommand,
  CreateTableCommand,
  GetTablesCommand,
  GlueClient,
} from "@aws-sdk/client-glue";


import { SimSdk } from "@kensio/yulin/sdk";


using simSdk = new SimSdk();
simSdk.intercept(GlueClient);


const client = new GlueClient({ region: "eu-west-2" });


await client.send(
  new CreateDatabaseCommand({ DatabaseInput: { Name: "site_logs" } }),
);
await client.send(
  new CreateTableCommand({
    DatabaseName: "site_logs",
    TableInput: { Name: "access_logs" },
  }),
);


const { TableList } = await client.send(
  new GetTablesCommand({ DatabaseName: "site_logs" }),
);


// access_logs
console.log(TableList?.[0]?.Name);

Permissions

Section titled “Permissions”

Every command uses simulated IAM. Catalog resources form a hierarchy. A table operation needs permission on the table, database and catalog. Deleting a database also needs permission on its tables.

/**
 * A Role that may read one table and nothing else in the catalog.
 */


import {
  CreateDatabaseCommand,
  CreateTableCommand,
  GetTableCommand,
} from "@aws-sdk/client-glue";
import { CreateRoleCommand, PutRolePolicyCommand } from "@aws-sdk/client-iam";


import { SimAws } from "@kensio/yulin";


const simAws = new SimAws({ defaultAccountId: "111111111111" });
const glue = simAws.glue();


glue.createDatabase(
  new CreateDatabaseCommand({ DatabaseInput: { Name: "site_logs" } }),
);
glue.createTable(
  new CreateTableCommand({
    DatabaseName: "site_logs",
    TableInput: { Name: "access_logs" },
  }),
);


await simAws.iam().createRole(
  new CreateRoleCommand({
    RoleName: "ReportingRole",
    AssumeRolePolicyDocument: JSON.stringify({
      Version: "2012-10-17",
      Statement: [
        {
          Effect: "Allow",
          Principal: { Service: "lambda.amazonaws.com" },
          Action: "sts:AssumeRole",
        },
      ],
    }),
  }),
);


await simAws.iam().putRolePolicy(
  new PutRolePolicyCommand({
    RoleName: "ReportingRole",
    PolicyName: "ReadAccessLogs",
    PolicyDocument: JSON.stringify({
      Version: "2012-10-17",
      Statement: [
        {
          Effect: "Allow",
          Action: "glue:GetTable",
          Resource: [
            "arn:aws:glue:us-east-1:111111111111:catalog",
            "arn:aws:glue:us-east-1:111111111111:database/site_logs",
            "arn:aws:glue:us-east-1:111111111111:table/site_logs/access_logs",
          ],
        },
      ],
    }),
  }),
);


const { Table } = glue.getTable(
  new GetTableCommand({ DatabaseName: "site_logs", Name: "access_logs" }),
  {
    caller: {
      kind: "arn",
      arn: "arn:aws:iam::111111111111:role/ReportingRole",
    },
  },
);


// access_logs
console.log(Table.Name);

A policy that grants only the table ARN is insufficient.

Available functionality

Section titled “Available functionality”
  • CreateDatabase, GetDatabase, GetDatabases and DeleteDatabase.
  • CreateTable, GetTable, GetTables and DeleteTable.
  • CreatePartition, BatchCreatePartition, GetPartition, GetPartitions, DeletePartition and BatchDeletePartition.
  • GetPartitions Expression filtering, over the SQL operators, AND, OR, NOT and brackets.
  • AWS::Glue::Database and AWS::Glue::Table, deployed and deleted with the stack.
  • TableInput.Parameters, PartitionKeys and StorageDescriptor, held and read back as declared.
  • IAM authorization on every Command, over the resource and its ancestors.
  • Database and table names folded to lower case, as the Data Catalog folds them.

Limitations

Section titled “Limitations”
  • Partition projection is held here and evaluated by Athena. The catalog stores the parameters as they were declared and materializes no partition from them. Simulated Athena expands them when a query runs, which is where a broken projection is refused.
  • The catalog reads no query and no object. SQL reaches Athena rather than the catalog, no S3 prefix is listed here, and no object is opened.
  • AWS::Glue::Crawler is absent. A crawler fills a catalog by reading objects, and every object stays unread here. A template declaring one deploys, with the crawler recorded on the stack’s skippedResources.
  • A registered partition is read by Athena, and a projected one wins over it. Simulated Athena reads a table’s registered partitions when a query runs, unless projection.enabled is true. Real Athena stops reading the catalog’s partitions once projection is on.
  • A GetPartitions Expression is the whole of the filtering. Segment and parallel listing, ExcludeColumnSchema and TransactionId are absent, and so are partition indexes, which change which expressions real Glue will serve rather than what any of them mean.
  • An expression compares a partition key against a literal. That is the whole grammar. Functions, arithmetic and comparing one column against another sit outside it, and a column name is written unquoted.
  • A partition keeps what it was registered with. UpdatePartition and BatchUpdatePartition are absent, along with BatchGetPartition and partition indexes.
  • AWS::Glue::Partition is absent. Real CloudFormation has the resource type, and a template declaring one deploys here with the partition recorded on the stack’s skippedResources. A stack that needs its partitions registered does it through the SDK once the deploy finishes.
  • A table keeps the definition it was created with. UpdateTable and UpdateDatabase are absent, and GetTable reports the creation time as the update time.
  • Fn::GetAtt Id on a table answers with a guess. It resolves to the catalog id, the database name and the table name joined with |, as in 111111111111|site_logs|access_logs. This is the one piece of behaviour here that nothing has checked against AWS. CloudFormation documents that the attribute exists and documents nothing about its value, so a template asserting on it agrees with this simulation and may disagree with a real deploy. Confirming it takes one stack deployed to an account with !GetAtt Table.Id as an output.
  • A lookup finds a name under any spelling. Real Glue documents the fold on the way in and asks the caller to pass lower case on the way back out. GetDatabase and GetTable fold the name they are given here, so a request naming Rainlytics reaches the database stored as rainlytics. This is the one piece of the fold nothing has checked against AWS. A test relying on it agrees with this simulation and may disagree with a real call, and passing the name in lower case avoids the question.
  • Cross-account catalogs are refused. A CatalogId naming another account is refused, in a Command and in a template.
  • Versions, statistics and Lake Formation are absent. A table has no version history and no column statistics, and IAM is the whole of the permission model.
  • Iceberg and other open table formats are recorded, never built. OpenTableFormatInput is reported as ignored and the table is created as an ordinary one.
  • Listings come back whole. GetDatabases, GetTables and GetPartitions answer with everything in creation order. MaxResults and NextToken are absent.
  • Connections, jobs, triggers, workflows and the Schema Registry are absent. The Data Catalog is the whole of the simulated surface.