This makes it easier to run distinct environments for say production and development, tests, or for different teams or security profiles. class DagBag ( BaseDagBag, LoggingMixin ): """ A dagbag is a collection of dags, parsed out of a folder tree and has high level configuration settings, like what database to use as a backend and what executor to use to fire off tasks. from _future_ import division from _future_ import unicode_literals import hashlib import imp import importlib import os import sys import textwrap import zipfile from collections import namedtuple from datetime import datetime, timedelta import six from croniter import croniter, CroniterBadCronError, CroniterBadDateError, CroniterNotAlphaError from sqlalchemy import or_ from airflow import configuration, settings from _dag import BaseDagBag from airflow.exceptions import AirflowDagCycleException from airflow.executors import get_default_executor from ttings import Stats from airflow.utils import timezone from _processing import list_py_file_paths, correct_maybe_zipped from import provide_session from import pprinttable from _mixin import LoggingMixin from import State from import timeout See the License for the # specific language governing permissions and limitations # under the License. You may obtain a copy of the License at # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License") you may not use this file except in compliance # with the License. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. Save attributes about list of DAG to the DB.# -*- coding: utf-8 -*- # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. Prints a report around DagBag loading stats sync_to_db ( processor_subdir = None, session = None ) ¶ collect_dags_from_db ( ) ¶Ĭollects DAGs from database. The DAG_IGNORE_FILE_SYNTAX configuration parameter. Un-anchored regexes or gitignore-like glob expressions, depending on Ignoring files that match any of the patterns specified The directory, it will behave much like a. airflowignore file is found while processing Imports them and adds them to the dagbag collection. Given a file path or a folder, this method looks for python modules, collect_dags ( dag_folder = None, only_if_updated = True, include_examples = conf.getboolean('core', 'LOAD_EXAMPLES'), safe_mode = conf.getboolean('core', 'DAG_DISCOVERY_SAFE_MODE') ) ¶ RaisesĪirflowDagDuplicatedIdException if this dag or its subdags already exists in the bag. RaisesĪirflowDagCycleException if a cycle is detected in this dag or its subdags. bag_dag ( dag, root_dag ) ¶Īdds the DAG into the bag, recurses into sub dags. The module and look for dag objects within it. Given a path to a python module or zip file, this method imports Gets the DAG out of the dictionary, and refreshes it if expired Parametersĭag_id – DAG Id process_file ( filepath, only_if_updated = True, safe_mode = True ) ¶ List get_dag ( dag_id, session = None ) ¶ Whether or not to read dags from DB property dag_ids ¶ ReturnsĪ list of DAG IDs in this bag Return type The amount of dags contained in this dagbag Return type Load_op_links ( bool) – Should the extra operator link be loaded via plugins whenĭe-serializing the DAG? This flag is set to False in Scheduler so that Extra Operator linksĪre not loaded to not run User code in Scheduler. If False DAGs are read from python files. Read_dags_from_db ( bool) – Read DAGs from DB if True is passed. Include_examples ( bool) – whether to include the examples that ship Settings are now dagbag level so that one system can run multiple,ĭag_folder ( str | pathlib.Path | None) – the folder to scan to find DAGs This makes it easier to runĭistinct environments for say production and development, tests, or forĭifferent teams or security profiles. Level configuration settings, like what database to use as a backend and DagBag ( dag_folder = None, include_examples = conf.getboolean('core', 'LOAD_EXAMPLES'), safe_mode = conf.getboolean('core', 'DAG_DISCOVERY_SAFE_MODE'), read_dags_from_db = False, store_serialized_dags = None, load_op_links = True ) ¶īases: _mixin.LoggingMixin Information about single file file :str ¶ duration :datetime.timedelta ¶ dag_num :int ¶ task_num :int ¶ dags :str ¶ class. A dagbag is a collection of dags, parsed out of a folder tree and has highĬlass.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |