Skip to content

Commit

Permalink
changed if man file
Browse files Browse the repository at this point in the history
  • Loading branch information
infinite-pursuits committed Oct 8, 2024
1 parent 40d8428 commit e8ffe47
Showing 1 changed file with 5 additions and 5 deletions.
10 changes: 5 additions & 5 deletions _posts/2024-10-07-ifman.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,27 +61,27 @@ This contribution is of independent technical interest, as the literature has on
All our experiments are on multi-class logistic regression models trained on ResNet50 embeddings for standard vision datasets. Our results are as follows.

<ol>
<li><strong>**Our Single-Target attack performs better than a non-influence Baseline.** Consider a non-influence baseline attack for increasing the importance of a training sample : reweigh the training loss, with a high weight on the loss for the target sample. Our attack has a significantly
<li><strong>**Our Single-Target attack performs better than a non-influence Baseline.** </strong>Consider a non-influence baseline attack for increasing the importance of a training sample : reweigh the training loss, with a high weight on the loss for the target sample. Our attack has a significantly
higher success rate as compared to the baseline with a much smaller accuracy drop under all
settings, as shown in the table below.</strong></li>
settings, as shown in the table below.</li>

<div class='l-body' align="center">
<img class="img-fluid rounded z-depth-1" src="{{ site.baseurl }}/assets/img/2024-10-ifman/baselinevsours.png">
<figcaption style="text-align: center; margin-top: 10px; margin-bottom: 10px;"> Success Rates of the Baseline vs. our Single-Target Attack for Data Valuation. $k$ is the ranking, as in top- $k$. ${\small \Delta_{\rm acc}}:= \small \rm TestAcc(\theta^*) - \small \rm TestAcc(\theta^\prime)$ represents drop in test accuracy for manipulated model $\theta^\prime$. Two success rates are reported : (1) when $\small \Delta_{\rm acc} \leq 3\%$ (2) the best success rate irrespective of accuracy drop. (\%) represents model accuracy. (-) means a model with non-zero success rate could not be found \& hence accuracy can't be stated. Our attack has a significantly higher success rate as compared to the baseline with a much smaller accuracy drop under all settings.</figcaption>
</div>


2. **Behavior of our Single-Target attack w.r.t manipulation radius $C$ & training set size.** Theoretically, the manipulation radius parameter $C$ in our attack objectives is expected to create a trade-off between the manipulated model's accuracy and the attack success rate. Increasing radius $C$ should result in a higher success rate as the manipulated model is allowed to diverge more from the (optimal) original model but on the other hand its accuracy should drop and vice-versa. We observe this trade-off for all three datasets and different values of ranking $k$, as shown in the figure below.
<li><strong>**Behavior of our Single-Target attack w.r.t manipulation radius $C$ & training set size.**</strong> Theoretically, the manipulation radius parameter $C$ in our attack objectives is expected to create a trade-off between the manipulated model's accuracy and the attack success rate. Increasing radius $C$ should result in a higher success rate as the manipulated model is allowed to diverge more from the (optimal) original model but on the other hand its accuracy should drop and vice-versa. We observe this trade-off for all three datasets and different values of ranking $k$, as shown in the figure below.

We also anticipate our attack to work better with smaller training sets, as there will be fewer samples competing for top- $k$ rankings. Experimentally, this is found to be true -- Pet dataset with the smallest training set has the highest success rates.
We also anticipate our attack to work better with smaller training sets, as there will be fewer samples competing for top- $k$ rankings. Experimentally, this is found to be true -- Pet dataset with the smallest training set has the highest success rates.</li>

<div class='l-body' align="center">
<img class="img-fluid rounded z-depth-1" src="{{ site.baseurl }}/assets/img/2024-10-ifman/singletarget.png">
<figcaption style="text-align: center; margin-top: 10px; margin-bottom: 10px;"> Behavior and Transfer results for Single-Target Attack in the Data Valuation use-case. Value of manipulation radius $C$ increases from left to right in each curve. (1) Behavior on original test set (solid lines) : As manipulation radius $C$ increases, manipulated model accuracy drops while attack success rate increases. (2) Transfer on an unknown test set (dashed lines): Success rate on an unknown test set gets better with increasing values of ranking $k$.</figcaption>
</div>


3. **Our attacks transfer when influence scores are computed with an unknown test set.** When an unknown test set is used to compute influence scores, our attacks perform better as ranking $k$ increases, as shown in the figure above. This occurs because rank of the target sample, optimized with the original test set, deteriorates with the unknown test set and a larger $k$ increases the likelihood of the target still being in the top-$k$ rankings.
<li><strong>**Our attacks transfer when influence scores are computed with an unknown test set.** </strong>When an unknown test set is used to compute influence scores, our attacks perform better as ranking $k$ increases, as shown in the figure above. This occurs because rank of the target sample, optimized with the original test set, deteriorates with the unknown test set and a larger $k$ increases the likelihood of the target still being in the top-$k$ rankings.</li>


4. **How does our Multi-Target Attack perform with changing target set size and desired ranking $k$?** Intuitively, our attack should perform better when the size of the target set is larger compared to ranking $k$ -- this is simply because a larger target set offers more candidates to take the top-$k$ rankings spots, thus increasing the chances of some of them making it to top- $k$. Our experimental results confirm this intuition; as demonstrated in the figure below, we observe that (1) for a fixed value of ranking $k$, a larger target set size leads to a higher success rate; target set size of $100$ has the highest success rates for all values of ranking $k$ across the board, and (2) the success rate decreases with increasing value of $k$ for all target set sizes and datasets. These results are for the high-accuracy similarity regime where the original and manipulated model accuracy differ by less than $3\%$.
Expand Down

0 comments on commit e8ffe47

Please sign in to comment.