Data Cataloging
The Data Catalog as a shared metastore
Ask "where does the schema live?" and on this exam the answer is almost always the AWS Glue Data Catalog. It is a centralized metadata repository organized into databases and tables, where each table records the schema, the S3 location, the partition keys, and runtime properties of one data set (Data discovery and cataloging in AWS Glue[1]). The catalog stores metadata only; your data stays in S3 (or RDS, Redshift, and so on). A table definition is a pointer plus a schema, not a copy of the data.
The single most testable property is that the Data Catalog is Apache Hive metastore compatible, so it serves as one shared metastore for multiple engines. Amazon Athena stores and queries table metadata in the Data Catalog using SQL, Amazon EMR accesses the same definitions for big-data processing, and Redshift Spectrum reads the same catalog to query S3 directly (Data Catalog integration[1]). Define a table once and all three engines see it. When a question describes Athena, EMR, and Spectrum all needing the same schema over the same S3 data, the design answer is one Glue Data Catalog table, not a separate definition per engine.
A classifier is the component that recognizes a data format (CSV, JSON, Parquet, ORC, Avro, and others) and infers a schema from it. Glue ships built-in classifiers and lets you add custom ones; a classifier is what produces the column names and types a crawler then writes into a table.
With the model fixed, the recurring decision is how the catalog gets populated: automatically by a crawler, or by hand. The next section covers crawlers; manual definition (CREATE TABLE in Athena, or CreateTable via API/IaC) is the right call for a small, stable schema where a crawler is pure overhead.
Crawlers: discover schema and partitions
A crawler is the automation that keeps the catalog current. It connects to a data store, runs a classifier to infer the schema, and then creates or updates the table and its partitions in the Data Catalog automatically (Using crawlers to populate the Data Catalog[1]). The win is that you do not hand-write DDL as data and its layout drift; the crawler discovers new columns and new partition folders for you. Reach for a crawler when the schema evolves or partitions arrive continuously; define the table by hand when it is small and fixed.
A crawler runs on demand or on a schedule that you express as a cron expression, choosing the frequency, days, and time of the run (Scheduling a crawler[2]). For large, frequently-updated data sets, listing every object on each run is slow and costly. The fix is Amazon S3 event mode: instead of listing the whole target, the crawler consumes S3 event notifications from an SQS queue and lists only the folders that changed between crawls, which makes recrawls faster and cheaper (Accelerating crawls using Amazon S3 event notifications[3]). The first crawl is always a full listing; only after that does the crawler run incrementally from events. When a stem stresses "large incremental data set, recrawl cost too high," S3 event mode is the intended answer.
To catalog something other than S3, the crawler needs an AWS Glue connection: a Data Catalog object that stores the connection information for a data store, including the JDBC URI, credentials, and VPC details, used by both crawlers and ETL jobs to reach the source or target (Adding a Glue connection[4]). Create a JDBC connection to crawl an Amazon RDS, Amazon Redshift, or on-premises database into the catalog; the connection is the reusable, secured handle, not something you re-enter per job.
Keeping partitions in sync: crawler vs MSCK vs projection
New files land in S3 under new partition folders, but a query returns nothing until the catalog knows those partitions exist. Three mechanisms close that gap, and the exam wants you to pick the right one rather than treat them as interchangeable.
Register partitions in the catalog: crawler or MSCK REPAIR TABLE
The traditional path writes partition entries into the Data Catalog. Re-running a crawler does this and also picks up schema changes. For Hive-style layouts you can instead run MSCK REPAIR TABLE in Athena: it scans the S3 file system for Hive-compatible (key=value) partitions added after the table was created, compares them to the catalog, and adds the missing ones (MSCK REPAIR TABLE[5]). Two limits matter. First, MSCK REPAIR TABLE only adds Hive-compatible partitions and only adds, never removes them; for non-Hive layouts, or to add a single partition or drop a stale one, use ALTER TABLE ADD PARTITION / DROP PARTITION instead. Second, it scans all subfolders and can time out on very large tables, so AWS recommends it for first-time setup or bulk reconciliation and ALTER TABLE ADD PARTITION for frequent incremental adds.
Skip the catalog lookup: partition projection
For highly partitioned tables, even fetching the partition list is the bottleneck. Normally Athena makes a GetPartitions call to the Data Catalog before pruning, and a table with very many partitions makes that call slow (Partition projection[6]). Partition projection removes the call: Athena calculates partition values and locations from table properties you configure, instead of doing a time-consuming metadata lookup, so an in-memory computation replaces a remote one and runtime drops on highly partitioned tables. It also automates partition management, because new date or integer partitions need no registration at all.
Projection has sharp edges to remember. Enabling it makes Athena ignore any partition metadata registered in the catalog for that table, so you do not mix projection with crawler/MSCK partitions on the same table. It fits predictable patterns: continuous integers, continuous dates, finite enumerated values, and AWS service log layouts. And it is Athena-only: if the same table is read through Redshift Spectrum, Athena for Spark, or EMR, those engines use standard catalog partitions, not projection. A query outside the projected range simply returns zero rows rather than an error.
Speed up the lookup you keep: Glue partition indexes
When you do keep catalog partitions, a partition index makes GetPartitions fetch a subset instead of loading every partition and filtering in memory (Creating partition indexes[7]). You can create a maximum of 3 partition indexes per table, and the index helps only when the first key of the index appears in the filter expression; the indexing solution supports the AND operator, while OR, IN, LIKE, and NOT sub-expressions are filtered after the index narrows the set. Unlike projection, partition indexes benefit Redshift Spectrum, EMR, and Glue ETL too, since they speed up the shared catalog rather than bypassing it.
Technical catalog vs business catalog
Two services are both called catalogs and they answer different questions, so the exam tests whether you can tell them apart. The Glue Data Catalog is the technical catalog: the Hive-compatible metastore that engines query against, holding schemas, locations, and partitions, covered in the sections above. Amazon SageMaker Catalog (formerly Amazon DataZone) is the business catalog: a data management service that makes it faster to catalog, discover, share, and govern data across AWS, on-premises, and third-party sources (What is Amazon DataZone?[8]). The mental split is simple: the technical catalog is for query engines, the business catalog is for people who need to find and request data.
The business catalog builds on the technical one rather than replacing it. With DataZone you publish data assets to the catalog from data already in the AWS Glue Data Catalog and from Amazon Redshift tables and views, and you can manually publish objects from S3 (DataZone integrations[8]). Analysts then search using business terms and business glossaries, request access through a governed subscription workflow, and analyze with Athena or Redshift query editors, while administrators govern access with fine-grained controls. So the data flows technical-to-business: Glue catalogs the bytes, DataZone makes them discoverable and governs who can subscribe. It is reached through the Amazon SageMaker management console (or the DataZone console) and a browser-based data portal.
Keep the boundary clean for answers. If a question is about how Athena or EMR resolves a table schema, that is the Glue Data Catalog. If it is about organization-wide data discovery, a business glossary, or a subscribe-and-approve access workflow, that is SageMaker Catalog / DataZone. Lake Formation fine-grained permissions, which both can rely on, are an authorization topic, not a cataloging one.
Exam-pattern recognition
Cataloging questions reward matching the stem's keyword to the one tool that fits, because the distractors are usually the other cataloging tools used wrongly.
"Same schema, many engines"
When Athena, EMR, and Redshift Spectrum must all query the same S3 data with one definition, the answer is a single AWS Glue Data Catalog table, leaning on its Hive-metastore compatibility. The wrong answers define the schema separately per engine.
"Schema/files keep changing, no manual DDL"
A Glue crawler on a cron schedule is the discovery tool. If the stem adds "data set is huge and recrawl cost/time is the problem," upgrade the answer to a crawler in S3 event mode, which lists only changed folders via SQS rather than the whole target. A plain scheduled crawler that re-lists everything is the tempting-but-costlier distractor.
"Hundreds of thousands of partitions, queries slow"
If the partitions follow a predictable date/integer/enum pattern, partition projection is the answer: it computes partitions from table properties and skips the GetPartitions call. If you must keep catalog partitions (for example, because Redshift Spectrum or EMR also reads the table), add a partition index instead, remembering the max of 3 per table and that the first index key must be in the filter. Choosing projection for a table other engines read is the trap, since projection is Athena-only.
"Register new partitions in the catalog"
For a bulk one-time reconciliation of Hive-style folders, MSCK REPAIR TABLE is correct; for frequent single additions, ALTER TABLE ADD PARTITION avoids the scan-everything timeout. MSCK on a non-Hive layout, or expecting it to remove deleted partitions, is the wrong choice.
"Catalog a JDBC database" / "discover data org-wide"
Cataloging an RDS, Redshift, or on-premises database needs an AWS Glue connection (JDBC URI, credentials, VPC) that a crawler or job reuses. A business glossary, organization-wide discovery, or a subscribe-and-approve access workflow points to Amazon SageMaker Catalog / DataZone (the business catalog), not the Glue Data Catalog (the technical one). Mixing those two up is the single most common cataloging distractor.
Keeping catalog partitions in sync: which tool when
| Concern | Glue crawler | MSCK REPAIR TABLE | Partition projection |
|---|---|---|---|
| Where partition values live | Written into the Data Catalog | Written into the Data Catalog | Computed at query time from table properties |
| Catalog GetPartitions lookup at query time | Yes | Yes | No (Athena skips it) |
| Schema/partition discovery | Infers schema and partitions automatically | Adds Hive-style partitions only; no schema change | No discovery; you define the value ranges |
| Best for | Drifting schema, ad-hoc or non-projectable layouts | Bulk-registering many new Hive partitions at once | Highly partitioned, predictably-named (date/int/enum) data |
| Works outside Athena | Yes (catalog is shared) | Yes (catalog is shared) | Athena-only; other engines use catalog partitions |
Decision tree
Sharp facts the exam loves — give these one last read before exam day.
Cheat sheet
Sharp facts the exam loves — scan these before test day.
Define a table once in the AWS Glue Data Catalog and Athena, Amazon EMR, and Redshift Spectrum all query it, because the catalog is Apache Hive metastore compatible. It stores metadata (database, table, schema, S3 location, partitions), not the data itself, so a table is a schema plus a pointer. When several engines must read the same S3 data with one definition, the answer is a single Glue Data Catalog table rather than a separate definition per engine.
Trap Defining the schema separately in each engine; the Hive-compatible catalog exists so one table definition serves Athena, EMR, and Spectrum at once.
- A crawler infers schema and writes tables and partitions for you
Point a Glue crawler at a data store and it runs a classifier to infer the schema, then creates or updates the table and its partitions in the Data Catalog automatically. It is the right tool when columns or partitions drift and you do not want to maintain DDL by hand. For a tiny, fixed schema a crawler is overhead; define the table once with CREATE TABLE or infrastructure-as-code instead.
- A classifier is what recognizes the format and infers columns
A crawler does not guess schema on its own; it applies a classifier that recognizes a data format such as CSV, JSON, Parquet, ORC, or Avro and produces the column names and types. Glue ships built-in classifiers and lets you add custom classifiers for formats the built-ins miss. The classifier is the schema-inference step inside the crawl, distinct from the crawler that schedules and writes to the catalog.
- Run crawlers on demand or on a cron schedule
A Glue crawler runs on demand or on a schedule you express as a cron expression, choosing frequency, days, and time. Use a schedule to keep the catalog current as new data lands without anyone triggering the crawl manually. There is no separate always-on listener for plain scheduled crawls; the schedule itself drives recrawls.
2 questions test this
- Use S3 event mode to recrawl only changed folders
When a large data set makes full recrawls slow and expensive, configure the crawler in Amazon S3 event mode: it consumes S3 event notifications from an SQS queue and lists only the folders that changed between crawls instead of the entire target, cutting recrawl time and cost. The first crawl is always a full listing; only subsequent crawls run incrementally from events. Reach for this when the stem stresses huge, incrementally-updated data and recrawl cost.
Trap Keeping a plain scheduled crawler that re-lists the whole bucket every run when the data set is huge; that is the costly option S3 event mode exists to replace.
- A Glue connection holds the JDBC details a crawler reuses
To catalog a database rather than S3, create an AWS Glue connection: a Data Catalog object that stores connection information (JDBC URI, credentials, VPC details) for a data store. Crawlers and ETL jobs both reach the source or target through it, so you configure the connection once and reuse it. Use a JDBC connection to crawl Amazon RDS, Amazon Redshift, or an on-premises database into the catalog.
- Partition projection computes partitions and skips the catalog lookup
For highly partitioned tables, Athena normally makes a GetPartitions call to the Data Catalog before pruning, and that call is slow when partitions number in the hundreds of thousands. Partition projection removes it: Athena calculates partition values and locations from table properties you configure, replacing a remote metadata lookup with an in-memory computation and dropping query runtime. It also automates partition management, because new date or integer partitions need no registration at all.
Trap Reaching for projection to cut query cost on a table with only a handful of partitions; the GetPartitions call it removes is only a bottleneck on highly partitioned tables.
- Projection fits predictable date, integer, and enum partitions
Partition projection works when partitions follow a predictable pattern: continuous integers, continuous dates or datetimes, a finite set of enumerated values (like Region or airport codes), or AWS service log layouts. You declare the value ranges and projection type per partition column in table properties. Irregular or unpredictable partition naming is not projectable, and a query for a value outside the configured range returns zero rows rather than an error.
- Enabling projection makes Athena ignore catalog partitions
Once partition projection is enabled on a table, Athena ignores any partition metadata registered for that table in the Glue Data Catalog or Hive metastore and projects values instead. So you do not mix projection with crawler- or MSCK-registered partitions on the same table; the projected configuration is the single source of truth. SHOW PARTITIONS also will not list projected partitions, because projection is a DML-only feature.
Trap Running a crawler or MSCK to add partitions to a table that already uses projection; Athena ignores those catalog entries, so the work has no effect.
- Partition projection is Athena-only
Partition projection is usable only when the table is queried through Athena. If the same table is read by Redshift Spectrum, Athena for Spark, or Amazon EMR, those engines fall back to standard catalog partition metadata, not projection. So when a table must serve engines beyond Athena, keep real catalog partitions (and consider a partition index) instead of relying on projection.
Trap Choosing partition projection for a table that Redshift Spectrum or EMR also reads; only Athena projects, so the other engines see no partitions unless they are in the catalog.
- MSCK REPAIR TABLE bulk-adds Hive-style partitions, never removes them
MSCK REPAIR TABLE scans the S3 file system for Hive-compatible (key=value) partition folders added after the table was created and adds the missing ones to the catalog. It only adds partitions and never removes them, so deleting folders in S3 and re-running it will not drop stale partitions; use ALTER TABLE DROP PARTITION for that. It also handles only Hive-style layouts; non-Hive layouts need ALTER TABLE ADD PARTITION.
Trap Expecting MSCK REPAIR TABLE to remove partitions whose S3 folders were deleted; it only adds Hive-style partitions, so stale entries linger until you DROP PARTITION.
- Use MSCK for bulk reconciliation, ALTER TABLE ADD for frequent single adds
MSCK REPAIR TABLE is best for first-time setup or a bulk reconciliation of many new Hive partitions at once, because it scans all subfolders and can time out on very large tables. For frequent incremental additions (such as a new daily partition) ALTER TABLE ADD PARTITION adds the partition directly without the scan-everything cost, avoiding the query timeouts MSCK hits when run repeatedly. Match the tool to add-once-in-bulk versus add-often-one-at-a-time.
Trap Running MSCK REPAIR TABLE on every new daily partition; it rescans all subfolders and times out, where ALTER TABLE ADD PARTITION adds just the one.
- A partition index speeds GetPartitions instead of bypassing it
When you keep catalog partitions, a Glue partition index lets GetPartitions fetch a subset of partitions rather than loading every partition and filtering in memory, which speeds queries as partition counts grow. Unlike projection, an index benefits Redshift Spectrum, EMR, and Glue ETL too, because it accelerates the shared catalog rather than skipping it. Reach for an index when many engines read a heavily-partitioned table and projection is off the table.
Trap Assuming a partition index is Athena-only like projection; the index speeds the shared catalog, so Spectrum, EMR, and Glue ETL benefit too.
5 questions test this
- A company stores log data in Amazon S3 partitioned by year, month, and day using Apache Hive-style partitions. The data is cataloged in AWS…
- A data engineering team manages an Amazon S3 data lake with over 2,000 partitions in several tables stored in the AWS Glue Data Catalog.…
- A data analytics team queries a large partitioned table in Amazon Athena that contains over 500,000 partitions organized by year, month,…
- A data engineering team is configuring an AWS Glue crawler to catalog Parquet files stored in Amazon S3. The S3 bucket contains millions of…
- A company is deploying a new data analytics platform and needs to organize metadata for hundreds of tables across multiple data sources in…
- Max 3 partition indexes per table, and the first key must be filtered
A table supports a maximum of 3 partition indexes, and an index only helps when the first key of the index appears in the filter expression. The indexing solution supports the AND operator; sub-expressions using OR, IN, LIKE, or NOT are ignored by the index and filtered afterward on the narrowed set. So design indexes around your real query patterns and lead the filter with the index's first key.
- Glue Data Catalog is the technical catalog; SageMaker Catalog is the business catalog
Two AWS services are both called catalogs and answer different questions. The Glue Data Catalog is the technical catalog: the metastore engines query for schema, location, and partitions. Amazon SageMaker Catalog (formerly Amazon DataZone) is the business catalog: it lets people catalog, discover, share, and govern data with business glossary terms across an organization. The technical catalog is for query engines, the business catalog is for people who need to find and request data.
Trap Pointing org-wide data discovery with a business glossary at the Glue Data Catalog; that is SageMaker Catalog / DataZone, which is the business catalog.
- SageMaker Catalog publishes assets from the Glue Data Catalog and Redshift
The business catalog builds on the technical one: in DataZone you publish data assets to the catalog from data already in the AWS Glue Data Catalog and from Amazon Redshift tables and views, and you can manually publish S3 objects. Analysts then search by business terms, request access through a governed subscribe workflow, and analyze with Athena or Redshift query editors. Data flows technical-to-business: Glue catalogs the bytes, DataZone makes them discoverable and governs who subscribes.
- Cataloging defines what data is, not who may read it
Crawlers, the Data Catalog, and partition tools answer where the data is and what its schema looks like. Deciding who may read which rows or columns is fine-grained access control, handled by Lake Formation, which is an authorization concern rather than a cataloging one. On a cataloging question, do not let a Lake Formation permissions distractor pull you off the metadata mechanics.
- Let a Glue ETL job update the catalog with enableUpdateCatalog instead of rerunning a crawler
Set enableUpdateCatalog to true and updateBehavior to UPDATE_IN_DATABASE when a Glue ETL job writes a DynamicFrame to a catalog table; the job then overwrites the schema and registers new partitions during the run, so new data is immediately queryable in Athena without a separate crawler.
Trap Scheduling a crawler after every ETL run to pick up partitions the job could have registered itself.
8 questions test this
- A company runs AWS Glue ETL jobs that transform data and write results to Amazon S3 partitioned by year, month, and day. The data engineer…
- A company has an AWS Glue ETL job that transforms data and writes to Amazon S3. The data schema occasionally evolves with new columns being…
- A data engineering team runs AWS Glue ETL jobs that transform data and write results to an Amazon S3 data lake. The team wants the ETL jobs…
- A company has an AWS Glue ETL job that transforms data from Amazon S3 and writes the output to another S3 location. The table schema in the…
- A data engineering team is using AWS Glue ETL jobs to transform data and write results to Amazon S3 with the metadata cataloged in the AWS…
- A company is developing an ETL pipeline using AWS Glue jobs that transform data and write results to Amazon S3. The transformed data often…
- A data engineering team runs AWS Glue ETL jobs that transform data and write to Amazon S3 in Parquet format. The source data schema…
- A company has an AWS Glue ETL job that processes incoming data and writes transformed results to Amazon S3 in Parquet format. The data…
- Use CRAWL_NEW_FOLDERS_ONLY to crawl only newly added partitions
Setting the crawler recrawl policy to CRAWL_NEW_FOLDERS_ONLY makes the first run a full scan and later runs process only new folders, cutting crawl time and cost for stable-schema tables that just gain partitions. This mode forces UpdateBehavior and DeleteBehavior to LOG, so a major schema change needs a temporary switch back to CRAWL_EVERYTHING.
5 questions test this
- A company has a rapidly growing data lake with new partitions added daily to partitioned tables in Amazon S3. Data engineers run AWS Glue…
- A company runs AWS Glue crawlers on a schedule to keep the Data Catalog synchronized with data stored in Amazon S3. The data has a stable…
- A company has a data lake stored in Amazon S3 with millions of objects organized in Hive-style partitions by year, month, and day. The data…
- A company stores daily transactional data in Amazon S3 using a partitioned folder structure organized by year, month, and day. The data…
- A company has configured an AWS Glue crawler with incremental crawls enabled to catalog data in an Amazon S3 data lake. After a major…
- Preserve hand-edited Glue tables with UpdateBehavior LOG plus MergeNewColumns
SchemaChangePolicy UpdateBehavior=LOG records detected schema changes without overwriting an existing table definition, while CrawlerOutput AddOrUpdateBehavior=MergeNewColumns adds only newly discovered columns and keeps manual edits. DeleteBehavior=DEPRECATE_IN_DATABASE marks missing tables deprecated, and Partitions AddOrUpdateBehavior=InheritFromTable makes partitions adopt the table schema to avoid HIVE_PARTITION_SCHEMA_MISMATCH.
15 questions test this
- A data engineering team manages a data lake where partitioned tables are cataloged using AWS Glue crawlers. The team noticed that some…
- A company uses AWS Glue crawlers to maintain an AWS Glue Data Catalog for their data lake. The company wants to prevent their production…
- A company has a data lake on Amazon S3 that receives daily data updates from multiple sources. The company uses AWS Glue crawlers to…
- A company uses AWS Glue crawlers to populate the AWS Glue Data Catalog with table metadata from Amazon S3. The data engineering team has…
- A data engineer is configuring an AWS Glue crawler to catalog data stored in Amazon S3. The data engineer needs to ensure that when the…
- A data engineering team uses AWS Glue crawlers to maintain the metadata for tables in the AWS Glue Data Catalog. The team has discovered…
- A company stores JSON files in Amazon S3 that are cataloged using an AWS Glue crawler. The data engineering team needs to implement a…
- A data engineering team manages a data lake with frequent file additions and deletions in Amazon S3. An AWS Glue crawler runs hourly to…
- A company uses an AWS Glue crawler to catalog data stored in Amazon S3. The data science team has manually refined the table schema in the…
- A company uses AWS Glue crawlers to discover schemas from CSV files stored in Amazon S3. The data engineering team notices that the crawler…
- A company has deployed an AWS Glue crawler to catalog data stored in Amazon S3. The data schema has been manually refined by the data…
- A company uses an AWS Glue crawler to catalog data stored in Amazon S3. The data schema is stable and rarely changes. When occasional…
- A data engineer is responsible for maintaining consistency between partitioned tables in Amazon S3 and their metadata in the AWS Glue Data…
- A company uses AWS Glue crawlers to automatically populate the AWS Glue Data Catalog with table schemas from data stored in Amazon S3. The…
- A company uses AWS Glue crawlers to maintain metadata for a data lake stored in Amazon S3. The data engineering team has manually modified…
- Glue archives a table version on every UpdateTable; GetTableVersions reads the history
By default UpdateTable always creates an archived version of the table before applying changes, and GetTableVersions / GetTableVersion retrieve that history for auditing and schema comparison. Pass SkipArchive=true to suppress version creation for routine metadata edits; crawlers also create versions when they apply a schema change.
Trap Building a custom change-log pipeline when the catalog already keeps versioned table history.
8 questions test this
- A data platform team maintains an AWS Glue Data Catalog that serves as a metadata repository for multiple analytics teams. The team…
- A data engineer maintains an AWS Glue Data Catalog with frequently changing table schemas. The engineer notices that every time the schema…
- A company has a data lake built on Amazon S3 with table metadata stored in the AWS Glue Data Catalog. Over time, crawlers have created…
- A data engineer needs to track all schema changes made to tables in the AWS Glue Data Catalog over time. The company requires the ability…
- A data engineering team is building an automated data pipeline that frequently updates table definitions in the AWS Glue Data Catalog using…
- A data engineer needs to implement an audit trail for schema changes in the AWS Glue Data Catalog. The organization requires the ability to…
- A data engineering team manages an AWS Glue Data Catalog that contains hundreds of tables. The team needs to implement a solution to track…
- A company maintains critical data tables in the AWS Glue Data Catalog and needs to track schema changes for compliance auditing. The…
- Lake Formation data filters give row- and column-level (cell) security
A Lake Formation data filter combines a row filter expression (for example product_type='pharma') with a column include or exclude list, and you grant SELECT with that filter to give cell-level security. Specifying all columns plus a row expression yields row-level only; including/excluding columns plus all rows yields column-level only; combining both yields cell-level. Filters apply to read operations, so only SELECT can carry a filter.
6 questions test this
- A financial services company uses AWS Lake Formation to manage a data lake containing customer information. The company has a compliance…
- A retail company has an Amazon S3 data lake with Apache Iceberg tables containing customer transaction data. Different analyst teams…
- A company has a data lake on Amazon S3 with tables cataloged in AWS Glue Data Catalog. The data engineering team must implement a security…
- A data engineering team is building an Amazon S3 data lake using Apache Iceberg tables managed through the AWS Glue Data Catalog. The team…
- A company has a data lake on Amazon S3 with metadata cataloged in the AWS Glue Data Catalog. The company uses AWS Lake Formation for…
- A healthcare company stores patient data in a data lake on Amazon S3. The company uses AWS Lake Formation to manage access controls. A data…
- Register a location in hybrid access mode so IAM and Lake Formation permissions coexist
Hybrid access mode lets existing IAM-based pipelines keep their access while you opt specific principals into Lake Formation fine-grained permissions on the same registered S3 location. It is the standard way to onboard new analyst teams to Lake Formation read access without disrupting existing ETL write workloads.
7 questions test this
- A healthcare company has existing data pipelines that write to Apache Iceberg tables in Amazon S3 using IAM policies for access control.…
- A data engineering team is configuring AWS Lake Formation to manage fine-grained access control for Apache Iceberg tables stored in Amazon…
- A company has an existing AWS Glue Data Catalog with tables that are currently accessed by ETL jobs using IAM policies for Amazon S3 and…
- A company is migrating from IAM-based access control to AWS Lake Formation permissions for their data lake. Currently, all IAM principals…
- A company is transitioning from AWS Glue IAM-based permissions to AWS Lake Formation permissions for their Data Catalog. Currently, all IAM…
- A company has an existing data lake on Amazon S3 with Apache Iceberg tables. The data engineering team manages write operations using…
- A company has an existing S3 data lake with Apache Iceberg tables that are managed by an ETL team using IAM policies for access control.…
- Use LF-Tags (LF-TBAC) to grant catalog access at scale
Lake Formation tag-based access control attaches LF-Tags (such as classification and domain) to databases and tables, then grants permissions via LF-Tag expressions. New resources tagged the same way are automatically covered, minimizing individual grants; cross-account LF-Tag sharing needs DESCRIBE and ASSOCIATE on the tags plus cross-account version 3 or higher.
Trap Granting table-by-table permissions to every new principal instead of matching on tags.
5 questions test this
- A financial services company has over 500 tables in their AWS Glue Data Catalog spread across multiple databases. The company needs to…
- A company is implementing a data lake governance strategy using AWS Lake Formation. The data lake contains hundreds of databases and…
- A large enterprise has hundreds of tables in their AWS Glue Data Catalog organized across multiple databases. The company frequently…
- A company has a centralized data lake account that needs to share Data Catalog databases and tables with multiple consumer AWS accounts in…
- A multinational company has separate AWS accounts for its US and European subsidiaries. The US account contains a data lake with customer…
- Creating a table on a registered location needs DATA_LOCATION_ACCESS
To create a Data Catalog table that points at an S3 location registered with Lake Formation, a principal needs both CREATE_TABLE on the database and DATA_LOCATION_ACCESS on that specific registered S3 location (or prefix). Registering the S3 location with a service-linked or custom IAM role is the prerequisite for any Lake Formation fine-grained access.
3 questions test this
- A company is building a centralized data catalog for their Amazon S3-based data lake using Apache Iceberg tables. The company wants to use…
- A data engineer is setting up AWS Lake Formation for a new data lake. The engineer has registered an Amazon S3 location with Lake Formation…
- A data engineer is configuring AWS Lake Formation for a new data lake project. The engineer needs to allow a data engineering team to…
- SageMaker Catalog catalogs and traces data and ML assets across the platform
Amazon SageMaker Catalog, built on the former Amazon DataZone, catalogs data and machine-learning assets and traces their lineage so teams can discover, govern, and understand the origin of assets across the platform. Where ML Lineage Tracking captures a single workflow's provenance graph, the catalog is the broader discovery-and-governance layer over many assets.