Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Re-encode for my csv problem #52

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Add --re-encode to allow importing files that are not in utf-8 (e.g. …
…a file in iso-8859-1/latin-1
scoates committed Dec 31, 2021
commit 2776b049a288eed1625f48facce4f466ac42fc02
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -273,6 +273,7 @@ If you have a column with a name such as `_commit_` it will be renamed too, addi
- `--namespace TEXT` - use this if you wish to include the history of multiple different files in the same database. The default is `item` but you can set it to something else, which will produce tables with names like `yournamespace` and `yournamespace_version`.
- `--wal` - Enable WAL mode on the created database file. Use this if you plan to run queries against the database while `git-history` is creating it.
- `--silent` - don't show the progress bar.
- `--re-encode TEXT` - re-encode the incoming data from this character encoding (e.g. `--re-encode=iso-8859-1` to convert from `Latin-1`); you can use `file -I filename` to get the encoding on some distributions.

### CSV and TSV data

11 changes: 10 additions & 1 deletion git_history/cli.py
Original file line number Diff line number Diff line change
@@ -122,6 +122,11 @@ def cli():
is_flag=True,
help="Don't show progress bar",
)
@click.option(
"--re-encode",
type=str,
help="Re-encodes incoming data from this charset (e.g. --re-encode=iso-8859-1)",
)
@click.version_option()
def file(
database,
@@ -143,6 +148,7 @@ def file(
wal,
debug,
silent,
re_encode,
):
"Analyze the history of a specific file and write it to SQLite"
if csv_ and convert:
@@ -231,7 +237,10 @@ def column_id(column):

# list() to resolve generators for repeated access later
try:
items = list(convert_function(content))
if re_encode:
items = list(convert_function(content.decode(re_encode).encode("utf-8")))
else:
items = list(convert_function(content))
except Exception:
print("\nError in commit: {}".format(git_hash))
raise