-
Notifications
You must be signed in to change notification settings - Fork 0
Ethics
Team have referred https://arxiv.org/pdf/1503.01817.pdf for getting many answers.
A.1 Informed consent: If there are human subjects, have they given informed consent, where subjects affirmatively opt-in and have a clear understanding of the data uses to which they consent?
All the dataset has creative common license
Each media object included in the dataset is represented by its metadata in the form its Flickr identifier, the user that created it, the camera that took it, the time at which it was taken and when it was uploaded, the location where it was taken (if available), and the CC license it was published under. In addition, the title, description and tags are also available, as well as direct links to its page and its content on Flickr.
To create the dataset, Author’s did not perform any specific filtering besides excluding photos and videos that had been marked as ‘screenshot’ or ‘other’ by the Flickr user.Author’s however, include as many videos as possible, because videos form a small percentage of media uploaded to Flickr and a random selection would have led to relatively few videos to be selected. Author’s further included as many photos as possible that were associated with a geographic coordinate to en- courage spatiotemporal research. Together these photos and videos form approximately half of the dataset, and the remainder is composed of CC photos randomly selected from the entire pool of photos on Flickr.
A.2 Collection bias: Have we considered sources of bias that could be introduced during data collection and survey design and taken steps to mitigate those?
Yes.The top 25 cameras used in the dataset are overwhelmingly digital single lens reflex (DSLR) models with the exception of the Apple iPhone. Considering that the most popular cameras in the Flickr community at the moment primarily consist of various iPhone models4, this bias in our data is likely due to CC licenses attracting a certain sub community of photographers that differs from the overall Flickr user base.
A.3 Limit PII exposure: Have we considered ways to minimize exposure of personally identifiable information (PII) for example through anonymization or not collecting information that isn't relevant for analysis?
Yes.Social features, comments, favorites, and followers/following data are not included in the dataset as, by their nature, these change on a day-to-day basis. They can, however, be easily obtained by querying the Flickr API.
A.4 Downstream bias mitigation: Have we considered ways to enable testing downstream results for biased outcomes (e.g., collecting data on protected group status like race or gender)?
Yes.Author’s used Caffe [11] to train 1,570 classifiers, each being a binary SVM, using 15 million photos taken from the entire Flickr corpus; positive examples were crowd labeled or hand- picked based on targeted search/group results, while nega- tive examples were drawn from a general pool. We tuned the classifiers such that they achieved at least 90% precision on a held-out test set.
B.1 Data security: Do we have a plan to protect and secure data (e.g., encryption at rest and in transit, access controls on internal users and third parties, access logs, and up-to-date software)?
Not exctly.Dataset has only CC license. However, Social features, comments, favorites, and followers/following data are not included in the dataset as, by their nature, these change on a day-to-day basis. They can, however, be easily obtained by querying the Flickr API.
B.2 Right to be forgotten: Do we have a mechanism through which an individual can request their personal information be removed?
Not Mentioned in the paper
B.3 Data retention plan: Is there a schedule or plan to delete the data after it is no longer needed?
Not Mentioned in the paper
C.1 Missing perspectives: Have we sought to address blind spots in the analysis through engagement with relevant stakeholders (e.g., checking assumptions and discussing implications with affected communities and subject matter experts)?
Yes.Flickr makes little distinction between a photo and a video; however, videos do play a role both on Flickr and in this dataset. Only 5% of the videos in our dataset do not have an audio track.
C.2 Dataset bias: Have we examined the data for possible sources of bias and taken steps to mitigate or address these biases (e.g., stereotype perpetuation, confirmation bias, imbalanced classes, or omitted confounding variables)?
Yes. Data has been taken from diverse source and it’s the largest data available.There are 68,552,616 photos and 418,507 videos in the dataset that have been annotated with user tags (or key- words). The tags make for a rich and diverse set of entities related to people (baby, family), animals (cat, dog), locations (park, beach), travel (nature, city), to name just the top few.
C.3 Honest representation: Are our visualizations, summary statistics, and reports designed to honestly represent the underlying data?
Not Mentioned in the paper
C.4 Privacy in analysis: Have we ensured that data with PII are not used or displayed unless necessary for the analysis?
Not Mentioned in the paper
C.5 Auditability: Is the process of generating the analysis well documented and reproducible if we discover issues in the future?
Yes.Each media object included in the dataset is represented by its metadata in the form its Flickr identifier, the user that created it, the camera that took it, the time at which it was taken and when it was uploaded, the location where it was taken (if available), and the CC license it was published under.
D.1 Proxy discrimination: Have we ensured that the model does not rely on variables or proxies for variables that are unfairly discriminatory?
Yes.To understand more about the visual content represented in the dataset, we used a deep learning approach to find the presence of a variety of concepts, such as people, animals, objects, food, events, architecture, and scenery.
D.2 Fairness across groups: Have we tested model results for fairness with respect to different affected groups (e.g., tested for disparate error rates)?
No
D.3 Metric selection: Have we considered the effects of optimizing for our defined metrics and considered additional metrics?
No
D.4 Explainability: Can we explain in understandable terms a decision the model made in cases where a justification is needed?
Need more exploration.
D.5 Communicate bias: Have we communicated the shortcomings, limitations, and biases of the model to relevant stakeholders in ways that can be generally understood?
Not Applicable as we are doing Academic Research
E.1 Redress: Have we discussed with our organization a plan for response if users are harmed by the results (e.g., how does the data science team evaluate these cases and update analysis and models to prevent future harm)?
Not Applicable as we are doing Academic Research
Yes
E.3 Concept drift: Do we test and monitor for concept drift to ensure the model remains fair over time?
We can see previous output to consider percentage error overtime.
E.4 Unintended use: Have we taken steps to identify and prevent unintended uses and abuse of the model and do we have a plan to monitor these once the model is deployed?
Yes