#aws#dataengineering
Organizing How to Use AWS Glue Workflow
Original Japanese article: AWS Glue Workflowの使い方について整理してみる Introduction I'm Aki, an AWS...
Aki9 min readd
Articles by @datapenguin
Browse the latest writing surfaced through DevArt.
Aki9 min read
#aws#dataengineering
Organizing the Use Cases of AWS Step Functions and Glue Workflow for ETL Processing with AWS Glue Jobs
Original Japanese article: Glue JobのETL処理におけるAWS Step FunctionsとGlue Workflowの使い分けを整理する ...
Aki7 min read
#aws#snowflake#dataengineering
Differences Between Snowflake Editions and Secure Connectivity with AWS
Original Japanese article: Snowflakeのエディションごとの違いとAWSとのセキュアな接続方法 Introduction I'm Aki,...
Aki8 min read
#aws#snowflake#dataengineering#iceberg
What Is Apache Polaris? Why Open Data Catalogs Matter and How to Use Them with AWS
Original Japanese article: Apache Polarisとは何か？オープンなデータカタログが求められる理由とAWSとの組み合わせ方を整理する ...
Aki11 min read
#aws#snowflake#dataengineering
Lightweight ETL on AWS Lambda Using DuckDB and Snowflake Connector
Original Japanese article: AWS Lambda × DuckDB × Snowflake ConnectorによるETLの実装 ...
Aki6 min read
#aws#snowflake#dataengineering
Securely Implementing Snowflake AWS Lambda Integration with Key Pair Authentication + Secrets Manager
Original Japanese article: Snowflake × AWS Lambda連携をKey Pair認証 + Secrets Managerで安全に実装する ...
Aki5 min read
#aws#snowflake#dataengineering#apacheiceberg
AWS Snowflake Lakehouse: 2 Practical Apache Iceberg Integration Patterns
Original Japanese article: AWSのレイクハウス（Apache Iceberg）をSnowflakeと連携する2つのパターンを整理する ...
Aki7 min read
#aws#dataengineering
Is AWS Glue Data Catalog Sufficient as a Data Catalog? Organizing Its Design, Limitations, and Complementary Strategies
Original Japanese article: AWS Glue Data Catalogはデータカタログとして十分か？設計・限界・補完戦略を整理する ...
Aki10 min read
#aws#duckdb#chdb#dataengineering
Boosting Lightweight ETL on AWS Lambda & Glue Python Shell with DuckDB and Apache Arrow Dataset
Original Japanese article: AWS Lambda/Glue Python Shell×DuckDBの軽量ETLをApache Arrow...
Aki9 min read
#aws#duckdb#chdb#iceberg
Does Increasing AWS Lambda Memory to 10GB Really Make It Faster? (AWS Lambda chDB/DuckDB PyIceberg Benchmark)
Original Japanese article: AWS Lambdaを10GBにすると本当に速くなるのか？（AWS Lambda×chDB/DuckDB×PyIceberg検証） ...
Aki7 min read
#aws#dataengineering#etl#lambda
S3 Triggers: How to Launch Glue Python Shell via AWS Lambda
Original Japanese article: S3トリガー×AWS Lambda×Glue Python Shellの起動パターン整理 ...
Aki8 min read
#aws#dataengineering
Exploring the Potential of AWS Glue Python Shell as a Long-Running Batch Execution Environment
Original Japanese article: AWS Glue Python Shellの長時間バッチ実行環境としての可能性を探る Introduction I'm...
Aki7 min read
#aws#dataengineering#etl
AWS Lambda and AWS Glue Python Shell in the Context of Lightweight ETL
Original Japanese article: 軽量ETLの文脈で考えるAWS LambdaとAWS Glue Python Shell ...
Aki7 min read
#aws#iceberg#duckdb#chdb
Lightweight ETL with AWS Glue Python Shell, chDB, and PyIceberg (Compared with DuckDB)
Original Japanese article: AWS Glue Python Shell×chDB×PyIcebergで実現する軽量ETLとDuckDB比較 ...
Aki8 min read
#aws#iceberg#chdb#duckdb
Lightweight ETL with AWS Lambda, chDB, and PyIceberg (Compared with DuckDB)
Original Japanese article: AWS Lambda×chDB×PyIcebergで実現する軽量ETLとDuckDB比較 ...
Aki5 min read
#aws
Looking at AWS Inter-Region Latency Through “Distance”
Original Japanese article: 「距離」から考えるAWSリージョン間通信 Introduction I'm Aki, an AWS Community...
Aki9 min read
#aws#dataengineering#data
Building a Data Platform on AWS: Essential Design Considerations for Power BI
Original Japanese article: AWSで構築するデータ基盤の基本とPower BI利用時の設計ポイント Introduction I'm Aki, an...
Aki5 min read
#aws#dataengineering#iceberg#lambda
Optimizing Data Processing on AWS with Data Compaction
Original Japanese article: AWSでの効率的なデータ処理を考える～データコンパクション～ Introduction I'm Aki, an AWS...
Aki7 min read
#aws#dataengineering#lambda#sqs
Designing a Cost-Efficient Parallel Data Pipeline on AWS Using Lambda and SQS
Original Japanese article: AWSでの効率的なデータ処理を考える～AWS LambdaとSQSを活用した並列データパイプライン～ ...
Aki6 min read
#aws#iceberg#deltalake#dataengineering
Interoperating Open Table Formats on AWS Using Apache XTable (Delta Iceberg)
Original Japanese article: Apache XTableを使ったAWS上でのOpen Table Format相互運用（Delta→Iceberg） ...
Aki4 min read
#aws#dataengineering#duckdb#iceberg
Dynamic Routing Lightweight ETL with AWS Lambda, DuckDB, and PyIceberg
Original Japanese article: AWS Lambda×DuckDB×PyIcebergで実現する動的ルーティング軽量ETLの実装 ...
Aki4 min read
#aws#dataengineering#delta
Lightweight ETL with AWS Lambda, DuckDB, and delta-rs
Original Japanese article: AWS Lambda×DuckDB×delta-rsによるETLの実装 Introduction I'm Aki, an...
Aki5 min read
#aws#duckdb#iceberg#dataengineering
Lightweight ETL with AWS Lambda, DuckDB, and PyIceberg
Original Japanese article: AWS Lambda×DuckDB×PyIcebergによるETLの実装 Introduction I'm Aki,...
Aki4 min read
#aws#iceberg#dataengineering
Build a Lightweight Serverless ETL Pipeline to Iceberg Tables with AWS Lambda Athena
Introduction I'm Aki, an AWS Community Builder (@jitepengin). Previously, I published two...
Aki4 min read
#aws#iceberg#dataengineering
PyIceberg on AWS Lambda: Comparing GlueCatalog and REST Catalog Access Methods
Original Japanese article: AWS Lambda × PyIceberg のカタログアクセスパターン比較 Introduction Apache...
Aki3 min read
#aws#iceberg#dataengineering#duckdb
Lightweight ETL with AWS Glue Python Shell, DuckDB, and PyIceberg
Introduction I'm Aki, an AWS Community Builder (@jitepengin). I previously posted an...
Aki7 min read
#aws
AWS IP Range Lookup Tool for Quick Security Checks
Introduction Security audits often require verifying whether a specific IP address belongs...
Aki5 min read
#aws#iceberg#opensource
Building a CLI Tool to Visualize AWS Iceberg Table Snapshot History
Introduction I'm Aki, an AWS Community Builder (@jitepengin). Apache Iceberg is one of...
Aki7 min read