Skip to content

Hf Hub

dataphy.sources.hf_hub

Functions

fetch(source: Dict, output_dir: str, include: str, runner: str = 'DirectRunner')

Source code in src/dataphy/sources/hf_hub.py 
 7
 8
 9
10
11
12
13
def fetch(source: Dict, output_dir: str, include: str, runner: str = "DirectRunner"):
    repo_id = source.get("repo_id")
    revision = source.get("revision", "main")
    local_dir = pathlib.Path(output_dir)
    local_dir.mkdir(parents=True, exist_ok=True)
    snapshot_download(repo_id=repo_id, revision=revision, local_dir=str(local_dir), local_dir_use_symlinks=False)
    return str(local_dir)