Changelog
Source:NEWS.md
microdatasus 3.0.0
Downloads
- Reworks
fetch_datasus()around an internal registry of supported systems and manifests built from the files actually published by DataSUS. Each relevant FTP directory is listed only once per call, avoiding redundant connections and URLs for files that do not exist. - Selects a single file for each system, period, state, and file part. Definitive/current data take precedence over preliminary data, and current data take precedence over historical copies. Results retain deterministic period, state, and file-part order.
- Uses
curldirectly for directory listings and downloads, with per-operation timeouts and up to two retries for transient network failures. Downloads remain sequential, display transfer progress by default, and no longer modify the globaloptions("timeout"). - Downloads through temporary partial files, validates file size and DBC contents, and reliably removes temporary files. With
stop_on_error = FALSE, successfully read files are returned after a consolidated warning; withstop_on_error = TRUE, any failure aborts the operation. - Applies
varsbefore accumulating results and combines files only once, reducing repeated copies. Whentrack_source = TRUE, thesourcecolumn is retained even whenvarsis supplied, and an existingsourcecolumn now produces a clear error. - Strengthens validation of years, months, states, logical arguments,
vars, and fractionaltimeoutvalues. Monthly systems require months, annual systems ignore them with one warning, and"all"can no longer be combined with individual states. - Replaces the former global 1996 lower-year restriction with historical limits specific to each information system. Existing public arguments, their order, and their defaults remain unchanged.
- Adds
quietat the end of thefetch_datasus()signature. Transfer progress and status messages, including each file name, are displayed by default; setquiet = TRUEto hide them. Warnings and errors remain visible. - Uses immediate
cliwarning alerts when month arguments or state selections are ignored, including whenquiet = TRUE. - Standardizes download messages across microdata and auxiliary tables, with consistent status verbs, semantic
cliformatting, indexed file progress, and one diagnostic item per failed directory or file. - Hardens
fetch_cadger()andfetch_sigtab()with shared timeout, retry, and temporary-file cleanup logic, and adds ZIP integrity and schema validation.
DBC support
- Removes the dependency on
read.dbcand reads DBC files with vendored C code adapted fromhealthbR. Thanks to Sidney Bissoli for the open-source code. - Adds
read_dbc()for local DBC files, with optional preservation of the column types inferred from DBF metadata. - Hardens
read_dbc()by rejecting directories and empty or unreadable input files before decompression, validating missing, unknown-size, and empty decompression output, and wrapping DBF read failures in contextual errors with stablemicrodatasus_dbc_errorsubclasses. - Expands local, CRAN-safe DBC tests for argument validation, type preservation, leading zeros, paths with spaces, repeat reads, temporary-file cleanup, and specific malformed header and compressed-stream errors.
Processing, documentation, and testing
- Adopts
dplyr::recode_values()becausedplyr::case_match()is deprecated. - Expands and corrects the documentation for all public functions, including network behavior, actual return types, compatibility arguments, and links to the corresponding chapters of the book Sistemas de Informação em Saúde no Brasil.
- Replaces skipped download tests with deterministic simulated listings and transfers. The suite now covers supported systems, historical, current, and preliminary files, multipart data, complete argument validation, retries, timeouts, empty listings and manifests, cleanup, corrupt and empty files, partial success,
vars,track_source, multiple states, and auxiliary-table schema validation. Localfile://fixtures exercisecurlwithout network access; live DataSUS smoke tests remain opt-in and are always skipped on CRAN. - Runs the active GitHub Actions R CMD check workflow for pushes and pull requests involving the
devbranch as well asmainandmaster. - Requires R 4.1.0 or later.
microdatasus 2.5.0
- Function
process_sinan_malaria, fix variable type when pre-processing the variableNU_IDADE_N(#132) - Function
process_sinasc, fix typo when pre-processing the variablePESO(#141). - Function
process_sia, fix typo when pre-processing the variablePA_MUNPCN(#137). - Function
process_cnesfor CNES-ST, fixMICR_REGandDISTRSANconversion. - Function
process_sih, fixNACIONALandHOMONIMOcodes, add moreFAEC_TPcodes (#131). - Remove
dtplyrpackage dependence.
microdatasus 2.4.2
- Fix error when processing data with
process_sinan_chikungunya().
microdatasus 2.4.1
- Fix typo in process functions that was changing variable names (
process_sim(),process_sinasc(),process_sinan_zika(),process_sinan_malaria(),process_sinan_leishmaniose_visceral(),process_sinan_leishmaniose_tegumentar(),process_sinan_dengue(),process_sinan_chikungunya(),process_sinan_chagas()).
microdatasus 2.4.0
- Update documentation on
fetch_datasus(). - Fetch SINAN-CHAGAS data.
- Process SINAN-CHAGAS data.
- Fetch SINAN-LEISHMANIOSE-VISCERAL and SINAN-LEISHMANIOSE-TEGUMENTAR data.
- Process SINAN-LEISHMANIOSE-VISCERAL and SINAN-LEISHMANIOSE-TEGUMENTAR data.
- Fetch SINAN-LEPTOSPIROSE data.
- Fix message when downloading data for all UFs.
microdatasus 2.3.4
- Fix error on tabNaturalidade, issue #123. Thanks @SophiaDamianoRovere and @hafermoraes for the suggestions.
- Fix FANTASIA names on CNES, issue #125.
microdatasus 2.3.1
CRAN release: 2024-06-18
- Import read.dbc package from CRAN.
- Remove CADGER data.
- General package adjustments for CRAN submission.
- Examples of process functions use data samples.
- Update package citation.
microdatasus 2.3.0
- All process functions uses dplyr verbs and dtplyr for fast processing.
- All process functions returns a tibble.
- Add
track_sourceargument tofetch_datasusfunction to create a variable calledsourcewith the file name that originated the row. - Updated CBO (CNES), CADGER (CNES), EQUIPE (SIA), SIGTAB (SIA) reference tables.
- Sample data for health information systems.
- Data files with XZ compression.
- Internal tests restructured.
- Progress bar removed from
process_simdue increased processing time.
microdatasus 2.2.7
- Enhance messages of
fetch_datasus. - Timeout the connection check when DataSUS FTP is down or not reachable.
- If there is no Internet or the DataSUS FTP is down or not reachable, the
fetch_datasuswill returnNULL. -
process_simshows a progress bar.
microdatasus 2.2.3
- As the package {read.dbc} is not available on CRAN, this patch uses the Github version.
- Issue #89 points out that files from SIM-EXT older than 2006 present 7-digits variable lengths for CODMUNRES and CODMUNOCOR. Those codes are now truncated to 6-digits standard by process_sim function.
- Related to #66, #84 and #86. Some files are very big to download, especially those from SIA and SIH. A timeout argument was added to the fetch_datasus with a default of 240 seconds.
- process_sinasc pull request #91 fixes issue #90, related to CODOCUPMAE variable
- Related to #79, now the function process_sia downloads an updated version from the SIGTAB table from DataSUS when nome_proced is TRUE.
microdatasus 2.2.1
- process_sinasc with new codes to process sex variable, avoiding missing results.
- process_cnes corrections to avoid NA introduction by coercion.
- process_sih corrects MUNIC_RES type
microdatasus 2.2.0
- fetch_datasus function uses read.dbc function with as.is set to TRUE for better performance.
- Minor bugs corrections.
microdatasus 2.1.0
- Bug correction that impacted download of all UFs from monthly data health systems.
microdatasus 1.4.4
- Updates SINAN functions for DataSUS changes in file structures, from per UF basis to all data (BR).
microdatasus 1.4.2
- Correct codmun handling for CNEST-ST data. Issue #38