chore(deps): update apache/airflow docker tag to v3.2.1#103
Open
renovate[bot] wants to merge 1 commit into
Open
chore(deps): update apache/airflow docker tag to v3.2.1#103renovate[bot] wants to merge 1 commit into
renovate[bot] wants to merge 1 commit into
Conversation
613cca6 to
a1a929e
Compare
a1a929e to
a045456
Compare
a045456 to
653418a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR contains the following updates:
3.1.6→3.2.1Release Notes
apache/airflow (apache/airflow)
v3.2.1Compare Source
Significant Changes
^^^^^^^^^^^^^^^^^^^
/dagsendpoint, as it now requires additional permissions (DagAccessEntity.RUN,DagAccessEntity.HITL_DETAIL, andDagAccessEntity.TASK_INSTANCE). This change was made because the endpoint returns aggregated data from these multiple entities. Please update your custom user roles to include read access for DAG Runs, Task Instances, and HITL Details if those users should still have access to the/dagsendpoint. (#64822)Improvements
^^^^^^^^^^^^
{}to restore OSS defaults. Thetokensfield is now optional in the theme configuration. (#64552)Bug Fixes
^^^^^^^^^
DEFAULT_LOGGING_CONFIGto use rightkwargs(#65412) (#65424)dispose_orm()not disposing async engine on shutdown (#65274) (#65284)get_team_name_depcreating wasted async sessions whenmulti_team=False(#65275) (#65282)disable_sqlite_fkeysto migration 0108 (#65288) (#65290)UPDATEto avoid row lock in the common case (#65029) (#65137)dropdownsin connection forms (#65007) (#65085) (#65138)SearchBarvalue not syncing withdefaultValuechanges (#65054) (#65140)$AIRFLOW_CONFIGenv (#64936) (#65200)Sessionstaying opened between yields (#65179) (#65195)Sessionleak fromStreamingResponseAPI endpoints (#65162) (#65193)_tokencookie exists from older Airflow instance (#64955) (#65177)@taskdecorator to validate operator arg types at decoration time (#65041) (#65050)is_alivedefault toNoneinjobs listCLI (#65065) (#65091)dag_idinget_task_instance(#64957) (#64968) (#65067)debounceon clear to prevent stale search value (#64893) (#64907)CommsDecoder(#64894) (#64946)UPDATEs insidedisable_sqlite_fkeysin migration 0097 (#64876) (#64940)TIexists in TIH (#61631) (#64693)SerializedDagModel(#64322) (#64738)TypeErrorinGET /dags/{dag_id}/taskswhenorder_byfield hasNonevalues (#64384) (#64587)DagRun(#64752) (#64853)connections importreturning non-zero exit code on failure (#64416) (#64449)targetand addrelattributes (#64542) (#64772)DagVersionSelectoptions not filtered by selectedDagRun(#64736) (#64771)start_datein example DAGs to avoid timezone conversion overflow (#63882) (#64758)AirflowPluginnot re-exported, causing mypy errors in plugins (#65132) (#65163)apache-airflow-providers-fabminimum version to preventconnexionimport error on Python 3.13 (#65523) (#65524)Miscellaneous
^^^^^^^^^^^^^
TriggerCommsDecodersync req-res cycle (#64882) (#65285)write_to_ossupport for writing task logs to OpenSearch (#64364) (#65201)airflow_local_settings.py(#64764) (#65003)Doc-only Changes
^^^^^^^^^^^^^^^^
v3.2.0Compare Source
Significant Changes
^^^^^^^^^^^^^^^^^^^
Asset Partitioning
""""""""""""""""""
The headline feature of Airflow 3.2.0 is asset partitioning — a major evolution of data-aware
scheduling. Instead of triggering Dags based on an entire asset, you can now schedule downstream
processing based on specific partitions of data. Only the relevant slice of data triggers downstream
work, making pipeline orchestration far more efficient and precise.
This matters when working with partitioned data lakes — date-partitioned S3 paths, Hive table
partitions, BigQuery table partitions, or any other partitioned data store. Previously, any update
to an asset triggered all downstream Dags regardless of which partition changed. Now only the right
work gets triggered at the right time.
For detailed usage instructions, see :doc:
/authoring-and-scheduling/assets.Multi-Team Deployments
""""""""""""""""""""""
Airflow 3.2 introduces multi-team support, allowing organizations to run multiple isolated teams within a single Airflow deployment.
Each team can have its own Dags, connections, variables, pools, and executors— enabling true resource and permission isolation without requiring separate Airflow instances per team.
This is particularly valuable for platform teams that serve multiple data engineering or data science teams from shared infrastructure, while maintaining strong boundaries between teams' resources and access.
For detailed usage instructions, see :doc:
/core-concepts/multi-team... warning::
Multi-Team Deployments are experimental in 3.2.0 and may change in future versions based on
user feedback.
Synchronous callback support for Deadline Alerts
""""""""""""""""""""""""""""""""""""""""""""""""
Deadline Alerts now support synchronous callbacks via
SyncCallbackin addition to the existingasynchronous
AsyncCallback. Synchronous callbacks are executed by the executor (rather thanthe triggerer), and can optionally target a specific executor via the
executorparameter.A Dag can also define multiple Deadline Alerts by passing a list to the
deadlineparameter,and each alert can use either callback type.
.. warning::
Deadline Alerts are experimental in 3.2.0 and may change in future versions based on
user feedback. Synchronous deadline callbacks (
SyncCallback) do not currentlysupport Connections stored in the Airflow metadata database.
For detailed usage instructions, see :doc:
/howto/deadline-alerts.UI Enhancements & Performance
"""""""""""""""""""""""""""""
Grid View Virtualization:
The Grid view now uses virtualization -- only visible rows are rendered to the DOM. This dramatically improves performance when viewing Dags with large numbers of task runs, reducing render time and memory usage for complex Dags. (#60241)
XCom Management in the UI:
You can now add, edit, and delete XCom values directly from the Airflow UI. This makes it much easier to debug and manage XCom state during development and day-to-day operations without needing CLI commands. (#58921)
HITL Detail History:
The Human-in-the-Loop approval interface now includes a full history view, letting operators and reviewers see the complete audit trail of approvals and rejections for any task. (#56760, #55952)
Gantt Chart Improvements:
New
--only-idleflag for the scheduler CLI"""""""""""""""""""""""""""""""""""""""""""""""
The
airflow schedulercommand has a new--only-idleflag that only counts runs when thescheduler is idle. This helps users run the scheduler once and process all triggered Dags and
queued tasks. It requires and complements the
--num-runsflag so one can set a small valueinstead of guessing how many iterations the scheduler needs.
Replace per-run TI summary requests with a single NDJSON stream
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
The grid, graph, gantt, and task-detail views now fetch task-instance
summaries through a single streaming HTTP request
(
GET /ui/grid/ti_summaries/{dag_id}?run_ids=...) instead of one requestper run. The server emits one JSON line per run as soon as that run's task
instances are ready, so columns appear progressively rather than all at once.
What changed:
GET /ui/grid/ti_summaries/{dag_id}?run_ids=...is now the sole endpointfor TI summaries, returning an
application/x-ndjsonstream where eachline is a serialized
GridTISummariesobject for one run.GET /ui/grid/ti_summaries/{dag_id}/{run_id}has been removed.
share the same
dag_version_id, avoiding redundant deserialization.task instance) use the stream endpoint, passing one or more
run_ids.Structured JSON logging for all API server output
"""""""""""""""""""""""""""""""""""""""""""""""""
The new
json_logsoption under the[logging]section makes Airflowproduce all its output as newline-delimited JSON (structured logs) instead of
human-readable formatted logs. This covers the API server (gunicorn/uvicorn),
including access logs, warnings, and unhandled exceptions.
Not all components support this yet — notably
airflow celery workerbutany non-JSON output when
json_logsis enabled will be treated as a bug. (#63365)Remove legacy OTel Trace metaclass and shared tracer wrappers
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
The interfaces and functions located in
airflow.traceswereinternal code that provided a standard way to manage spans in
internal Airflow code. They were not intended as user-facing code
and were never documented. They are no longer needed so we
remove them in 3.2. (#63452)
Move task-level exception imports into the Task SDK
"""""""""""""""""""""""""""""""""""""""""""""""""""
Airflow now sources task-facing exceptions (
AirflowSkipException,TaskDeferred, etc.) fromairflow.sdk.exceptions.airflow.exceptionsstill exposes the same exceptions, but they areproxies that emit
DeprecatedImportWarningso Dag authors can migrate before the shim is removed.What changed:
airflow-coreat runtime.airflow.providers.common.compat.sdkcentralizes compatibility imports for providers.Behaviour changes:
ValueError(instead ofAirflowException) whenpoke_interval/timeoutarguments are invalid.airflow.exceptionslogs a warning directing users tothe SDK import path.
Exceptions now provided by
airflow.sdk.exceptions:AirflowExceptionandAirflowNotFoundExceptionAirflowRescheduleExceptionandAirflowSensorTimeoutAirflowSkipException,AirflowFailException,AirflowTaskTimeout,AirflowTaskTerminatedTaskDeferred,TaskDeferralTimeout,TaskDeferralErrorDagRunTriggerExceptionandDownstreamTasksSkippedAirflowDagCycleExceptionandAirflowInactiveAssetInInletOrOutletExceptionParamValidationError,DuplicateTaskIdFound,TaskAlreadyInTaskGroup,TaskNotFound,XComNotFoundAirflowOptionalProviderFeatureExceptionBackward compatibility:
airflow.exceptionscontinue to work, thoughthey log warnings.
airflow.providers.common.compat.sdkto keep one import path that worksacross supported Airflow versions.
Migration:
airflow.sdk.exceptions(or from the provider compat shim).ValueErrorfor invalid sensor arguments if itpreviously caught
AirflowException.Support numeric multiplier values for retry_exponential_backoff parameter
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
The
retry_exponential_backoffparameter now accepts numeric values to specify custom exponential backoff multipliers for task retries. Previously, this parameter only accepted boolean values (TrueorFalse), withTrueusing a hardcoded multiplier of2.0.New behavior:
2.0,3.5) directly specify the exponential backoff multiplierretry_exponential_backoff=2.0doubles the delay between each retry attemptretry_exponential_backoff=0orFalsedisables exponential backoff (uses fixedretry_delay)Backwards compatibility:
Existing Dags using boolean values continue to work:
retry_exponential_backoff=True→ converted to2.0(maintains original behavior)retry_exponential_backoff=False→ converted to0.0(no exponential backoff)API changes:
The REST API schema for
retry_exponential_backoffhas changed fromtype: booleantotype: number. API clients must use numeric values (boolean values will be rejected).Migration:
While boolean values in Python Dags are automatically converted for backwards compatibility, we recommend updating to explicit numeric values for clarity:
retry_exponential_backoff=True→retry_exponential_backoff=2.0retry_exponential_backoff=False→retry_exponential_backoff=0Move serialization/deserialization (serde) logic into Task SDK
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
Airflow now sources serde logic from
airflow.sdk.serdeinstead ofairflow.serialization.serde. Serializer modules have moved fromairflow.serialization.serializers.*to
airflow.sdk.serde.serializers.*. The old import paths still work but emitDeprecatedImportWarningto guide migration. The backward compatibility layer will be removed in Airflow 4.
What changed:
airflow-coretotask-sdkpackageairflow.serialization.serializers.*toairflow.sdk.serde.serializers.*airflow.sdk.serde.serializers.*namespaceCode interface changes:
airflow.sdk.serde.serializers.*instead ofairflow.serialization.serializers.*airflow.sdk.serdeinstead ofairflow.serialization.serdeBackward compatibility:
airflow.serialization.serializers.*continue to work with deprecation warningsMigration:
airflow.sdk.serde.serializers.*airflow.sdk.serde.serializers.*namespace (e.g., createtask-sdk/src/airflow/sdk/serde/serializers/your_serializer.py)Methods removed from PriorityWeightStrategy
""""""""""""""""""""""""""""""""""""""""""""
On (experimental) class
PriorityWeightStrategy, functionsserialize()and
deserialize()were never used anywhere, and have been removed. Theyshould not be relied on in user code. (#59780)
Methods removed from TaskInstance
"""""""""""""""""""""""""""""""""
On class
TaskInstance, functionsrun(),render_templates(),get_template_context(), and private members related to them have beenremoved. The class has been considered internal since 3.0, and should not be
relied on in user code. (#59780, #59835)
Modify the information returned by
DagBag"""""""""""""""""""""""""""""""""""""""""""""
New behavior:
DagBagnow usesPath.relative_tofor consistent cross-platform behavior.FileLoadStatnow has two additional nullable fields:bundle_pathandbundle_name.Backward compatibility:
FileLoadStatwill no longer produce paths beginning with/with the meaning of "relative to the dags folder".This is a breaking change for any custom code that performs string-based path manipulations relying on this behavior.
Users are advised to update such code to use
pathlib.Path. (#59785)Remove
--conn-idoption fromairflow connections list"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
The redundant
--conn-idoption has been removed from theairflow connections listCLI command.Use
airflow connections getinstead. (#59855)Add operator-level
render_template_as_native_objoverride"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
Operators can now override the Dag-level
render_template_as_native_objsetting,enabling fine-grained control over whether templates are rendered as native Python
types or strings on a per-task basis. Set
render_template_as_native_obj=TrueorFalseon any operator to override the Dag setting, or leave asNone(default)to inherit from the Dag.
Add gunicorn support for API server with zero-downtime worker recycling
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
The API server now supports gunicorn as an alternative server with rolling worker restarts
to prevent memory accumulation in long-running processes.
Key Benefits:
Rolling worker restarts: New workers spawn and pass health checks before old workers
are killed, ensuring zero downtime during worker recycling.
Memory sharing: Gunicorn uses preload + fork, so workers share memory via
copy-on-write. This significantly reduces total memory usage compared to uvicorn's
multiprocess mode where each worker loads everything independently.
Correct FIFO signal handling: Gunicorn's SIGTTOU kills the oldest worker (FIFO),
not the newest (LIFO), which is correct for rolling restarts.
Configuration:
.. code-block:: ini
Use gunicorn instead of uvicorn
Enable rolling worker restarts every 12 hours
Restart workers one at a time
Or via environment variables:
.. code-block:: bash
Requirements:
Install the gunicorn extra:
pip install 'apache-airflow-core[gunicorn]'Note on uvicorn (default):
The default uvicorn mode does not support rolling worker restarts because:
If you need worker recycling or memory-efficient multi-worker deployment, use gunicorn. (#60921)
Improved performance of rendered task instance fields cleanup for Dags with many mapped tasks (~42x faster)
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
The config
max_num_rendered_ti_fields_per_taskis renamed tonum_dag_runs_to_retain_rendered_fields(old name still works with deprecation warning).
Retention is now based on the N most recent dag runs rather than N most recent task executions,
which may result in fewer records retained for conditional/sparse tasks. (#60951)
AuthManager Backfill permissions are now handled by the
requires_access_dagon theDagAccessEntity.Run""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
is_authorized_backfillof theBaseAuthManagerinterface has been removed. Core will no longer call this method and theirprovider counterpart implementation will be marked as deprecated.
Permissions for backfill operations are now checked against the
DagAccessEntity.Runpermission using the existingrequires_access_dagdecorator. In other words, if a user has permission to run a Dag, they can perform backfill operations on it.Please update your security policies to ensure that users who need to perform backfill operations have the appropriate
DagAccessEntity.Runpermissions. (Usershaving the Backfill permissions without having the DagRun ones will no longer be able to perform backfill operations without any update)
Python 3.14 support added
"""""""""""""""""""""""""
Airflow 3.2.0 adds support for Python 3.14. (#63787)
Reduce API server memory by eliminating
SerializedDAGloads on task start"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
The API server no longer loads the full
SerializedDAGwhen starting tasks,significantly reducing memory usage. (#60803)
Remove MySQL client from container images
"""""""""""""""""""""""""""""""""""""""""
MySQL client support has been removed from official Airflow container images. MySQL users
building on official images must install the client themselves. (#57146)
Add support for async callables in PythonOperator
"""""""""""""""""""""""""""""""""""""""""""""""""
The
PythonOperatorparameterpython_callablenow also supports async callables in Airflow 3.2,allowing users to run async def functions without manually managing an event loop. (#60268)
Make start_date optional for @continuous schedule
"""""""""""""""""""""""""""""""""""""""""""""""""
The
schedule="@​continuous"parameter now works without requiring astart_date, and any Dags with this schedule will begin running immediately when unpaused. (#61405)Upgrade SQLAlchemy (SQLA) to 2.0
"""""""""""""""""""""""""""""""""""""""""""""""""
Airflow now only support SQLAlchemy version 2
New Features
^^^^^^^^^^^^
PYTHON_LTObuild argument (#58337)--queuesCLI option for thetriggercommand (#59239)--show-valuesand--hide-sensitiveflags to CLIconnections listandvariables listto hide sensitive values by default (#62344)AIRFLOW__SECRETS__BACKEND_KWARG__<KEY>environment variables (#63312)only_newparameter to Dag clear to only clear newly added task instances (#59764)log_timestamp_formatconfig option for customizing component log timestamps (#63321)--action-on-existing-keyoption topools importandconnections importCLI commands (#62702)--use-migration-filesflag forairflow db init(#62234)AllowedKeyMapperfor partition key validation in asset partitioning (#61931)ChainMapperfor chaining multiple partition mappers (#64094)AgenticOperator(#63081)@task.stubdecorator to allow tasks in other languages to be defined in Dags (#56055)TriggerDagRunOperator(#60810)allowed_run_typesto whitelist specific Dag run types (#61833)dag_idanddag_run_idin bulk task instance endpoint (#57441)operator_name_pattern,pool_pattern,queue_patternas task instance search filters (#57571)update_masksupport for bulk PATCH APIs (#54597)sourceparameter toParam(#58615)TaskInstanceonRuntimeTaskInstance(#59712)max_trigger_to_select_per_loopconfig for Triggerer HA setup (#58803)uvicorn_logging_levelconfig option to control API server access logs (#56062)executor.running_dagsgauge metric to expose count of running Dags (#52815)GitDagBundle(#59911)GitHookfor Dag bundles (#58194)RemoteIOforObjectStorage(#54813)--devflag (#57741)auth list-envscommand to list CLI environments and auth status (#61426)airflow infocommand output (#59124)db_cleanto explicitly include or exclude Dags (#56663)TaskStreamFilter(#60549)globalCssin custom themes (#61161)run_afterdate filter on Dag runs page (#62797)AIRFLOW__API__THEMEin addition to brand (#64232)Bug Fixes
^^^^^^^^^
non-sensitive-onlyvalue asTrue(#59880)InvalidStatsNameExceptionfor pool names with invalid characters by auto-normalizing them when emitting metrics (#59938)AIRFLOW__API__BASE_URLbasename is configured (#63141)/mappedto group URLs (#63205)ti_skip_downstreamoverwriting RUNNING tasks to SKIPPED in HA deployments (#63266)@taskdecorator failing for tasks that return falsy values like0or empty string (#63788)LatestOnlyOperatornot working when direct upstream of a dynamically mapped task (#62287)XComreturn type in mapped task groups with dynamic mapping (#59104)DagRunspan emission crash whencontext_carrierisNone(#64087)next_dagrunfields areNone(#63962)relativedelta(#61671)task_instance_mutation_hookreceivingrun_id=NoneduringTaskInstancecreation (#63049)Nonedag_versionaccess (#62225)MetastoreBackend.expunge_all()corrupting shared session state (#63080)airflowignorenegation pattern handling for directory-only patterns (#62860)TYPE_CHECKING-only forward references in TaskFlow decorators (#63053)structlogJSON serialization crash on non-serializable objects (#62656)queued_taskstype mismatch in hybrid executors (CeleryKubernetesExecutor,LocalKubernetesExecutor) (#63744)pathlib.Pathobjects incorrectly resolved by Jinja templater in Task SDK (#63306)make_partial_modelfor API Pydantic models (#63716)_execution_api_server_url()ignoring configured value and falling back to edge config (#63192)DetachedInstanceErrorforairflow tasks rendercommand (#63916)@taskdefinition causing Dag parsing errors (#62174)limitparameter not sent inexecute_listserver requests (#63048)airflow.configurationcausingImportErroron Python 3.14 (#63787)map_indexrange validation in CLI commands (#62626)FabAuthManagerrace condition on startup with multiple workers (#62737)FabAuthManagerrace condition when workers concurrently create permissions, roles, and resources (#63842)JWTValidatornot handling GUESS algorithm with JWKS (#63115)FabAuthManagerfirst idle MySQL disconnect in token auth (#62919)JWTBearerTIPathDepimport errors in Human-In-The-Loop routes (#63277)log_pos(#63531)nulldag_run_confcausing serialization error inBackfillResponse(#63259)savepointswith per-Dag transactions (#63591)deadline.callback_id(#63612)intervalcausing query failures indeadline_alert(#63494)serialize_dagquery failure during deadline migration (#63804)visibility_timeoutthat kills long-running Celery tasks (#62869)CronPartitionedTimetable(#62441)AssetModelwhen updating asset partition DagRun — adds mutex lock (#59183)auth_managerload_usercausingPendingRollbackError(#61943)joinedloadfor asset indags_needing_dagruns()(#60957)NotMappedexception when clearing task instances with downstream/upstream (#58922)partition_keyfilter inPALKwhen creating DagRun (#61831)ObjectStoragePathto excludeconn_idfrom storage options passed to fsspec (#62701)XComObjectStorageBackend(#55805)default_email_on_failure/default_email_on_retryfrom config (#59912)TaskInstance.get_dagrunreturning None intask_instance_mutation_hook(#60726)dag_display_nameproperty bypass forDagStatsquery (#64256)TaskAlreadyRunningErrornot raised when starting an already-running task instance (#60855)enable_swagger_uiconfig not respected in API server (#64376)conf.has_optionnot respects default provider metadata (#64209)TaskInstancecrash when refreshing task weight for non-serialized operators (#64557)XComedit modal value not repopulating on reopen (#62798)RenderedJsonFieldcollapse behavior (#63831)RenderedJsonFieldnot displaying in table cells (#63245)-1) slots not rendering correctly (#62831)DurationChartlabels and disable animation flicker during auto-refresh (#62835)/dagrunspage not working (#62848)total_receivedcount in partitioned Dag runs view (#62786)Configuration
📅 Schedule: (UTC)
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.
This PR was generated by Mend Renovate. View the repository job log.